Repository: dbt-labs/dbt-core

Issue:
DBT cli throw memory exception for ffi.callback() on Mac with Apple Sillicon
### Describe the bug

`dbt run` or `dbt --version` throw error `Cannot allocate write+execute memory for ffi.callback().`

### Steps To Reproduce

On Macbook pro with M1 chip
`brew tap fishtown-analytics/dbt`
`brew install dbt`
add default profile.yml with connection to snowflake
create new project `dbt init`
`dbt run` with the two sample models, or simply `dbt --version`

### Expected behavior

Should complete successfully, instead of throwing exception. The process works fine on Intel-based Macbook pro.

### Screenshots and log output
```shell
> dbt run
Running with dbt=0.19.0
Found 3 models, 8 tests, 0 snapshots, 0 analyses, 143 macros, 0 operations, 0 seed files, 0 sources, 0 exposures
Encountered an error:
Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
```

### System information
**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [ ] bigquery
- [x] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
Encountered an error:
Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
```

**The operating system you're using:**
macOS Big Sur 11.2.2

**The output of `python --version`:**
3.8.7

### Additional context
Add any other context about the problem here.



Discussion:
Hey @alyiwang, thanks for the issue! We've been hearing about plenty of installation errors with M1 macs, though this is the first I'm seeing of this specific error. Unfortunately, I don't have access to the hardware to reproduce and debug this, so I'm operating on a mix of intuition and what I can find online.

It looks like this error is coming from `pyOpenSSL` (https://github.com/pyca/pyopenssl/issues/873), which is a dependency of [`snowflake-connector-python`](https://github.com/snowflakedb/snowflake-connector-python/blob/29ad620f7d8bb14f2bb0bc1d95b4dc35b65447bb/setup.py#L201). Other CLI tools that depend on `pyOpenSSL` are encountering the same stumbling block on M1 chips (e.g. https://github.com/Azure/azure-cli/issues/16907). If this is indeed the cause, I don't think there's anything we can do until either pyOpenSSL finds a resolution or the Snowflake connector removes it as a dependency.

In the meantime, you could try:
- `pip install dbt-snowflake` (N.B. this requires you to use python 3.8, since the pinned version of the Snowflake connector does not yet support py39)
- Install dbt via Homebrew using Rosetta ([SO post](https://stackoverflow.com/questions/64882584/how-to-run-the-homebrew-installer-under-rosetta-2-on-m1-macbook/64883440))
Hey @alyiwang, I was having similar issues and I was able to work through it using the following:

- First, follow this link to alter your terminal of choice (I was using iterm) to https://alexslobodnik.medium.com/apple-m1-python-pandas-and-homebrew-20f14828ccc7 open using Rosetta
- Uninstall all versions of dbt through `brew uninstall dbt` if you already tried installing
- I performed the above until `which dbt` returned nothing (at this point I had multiple installs of dbt that I had to get rid of
- Once my system was rid of dbt, I shifted over to the python virtual environment route.  Using Python 3.8, I ran `pip install virtualenv` and created a virtual environment.  Link for more info on those https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
- After activating the environment I ran `pip install dbt==0.18.0` and let it run it's course.  Following installation I ran `dbt --version` while the environment was still active and it worked.  It's worth noting that if you go this route of installation, the virtual environment must first be activated before running any dbt commands and deactivated when complete
Hi @jtcohen6 and @mikelanzago , thanks for your responses! I spent quite some time on this issue, and as you mentioned, it is a dependency issue with snowflake-connector-python, more specifically, it's some of it's dependencies, like pyOpenSSL and pyarrow, that's having compatibility issue with M1 chip. I was able to install latest pyOpenssl via pip, but the installation of snowflake-connector-python failed by pyarrow error. After many try and fail, I got `pip install snowflake-connector-python` (v2.4.1) to pass, which involves installing latest pyarrow through conda-forge https://anaconda.org/conda-forge/pyarrow which has osx-arm64 version. Then I realized dbt is fixed to an earlier version 2.3.5 of snowflake, which in turn requires an earlier version of pyarrow that doesn't have arm version anywhere. 

To make the installation work, a couple changes will need to be made from dbt to:
1. allow python3.9
2. use latest version 2.4.1 of snowflake-connector-python
@jtcohen6 do you think this can be put into the roadmap or feature request of DBT?

Meanwhile, I got the docker image version of DBT to work locally, even though I don't prefer this way of installation, this is the only thing working for me to unblock my work. I will definitely try out what Mike detailed later today. Thanks!
> To make the installation work, a couple changes will need to be made from dbt to:
>
> 1. allow python3.9
> 2. use latest version 2.4.1 of snowflake-connector-python
> @jtcohen6 do you think this can be put into the roadmap or feature request of DBT?

@alyiwang This is absolutely our plan for the next minor version of dbt (v0.20.0). `dbt-core` already supports python3.9, it's just a matter of upgrading our pin on the Snowflake connector.
Just ran into the issue as well today on my new work computer. If there is anything I can do to help out I'd be happy to chip in. The `snowflake-python-connector` continues to be my biggest pain in the behind with their dependency management.
We've just run headfirst into this issue with a colleague using a new macbook 😂 ; any rough timelines for a v0.20.0 pre-release @jtcohen6?  
Same issue here guys, looking forward to this new release to fix my local setup.
Workaround installing though pip following this thread: https://github.com/psycopg/psycopg2/issues/1200
Finally worked here!
I found this link to be helpful: https://nhancv.medium.com/macbook-pro-m1-homebrew-install-error-7170e4816894
**TL;DR**: if you experience the error during `dbt run`, just add those lines to `~/.dbt/profiles.yml` (see the full doc in [profiles.yml here](https://docs.getdbt.com/reference/profiles.yml))

```yaml
config:
    send_anonymous_usage_stats: False
```

**Details**:

Here's the stack trace (`MacBook Air (M1, 2020)` running `Mac OS 11.3.1`). The exception is raised in user behavior tracking code. Quickfix will be disabling tracking (see above). I guess the proper way to handle this would be wrapping `tracker.flush()` with `try/except`. Failure to track isn't a critical error after all 

```
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 125, in main
    results, succeeded = handle_and_check(args)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 203, in handle_and_check
    task, res = run_from_args(parsed)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 256, in run_from_args
    results = task.run()
  File "/opt/homebrew/Cellar/python@3.8/3.8.10/Frameworks/Python.framework/Versions/3.8/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 229, in track_run
    dbt.tracking.flush()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/tracking.py", line 428, in flush
    tracker.flush()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/tracker.py", line 218, in flush
    emitter.sync_flush()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/emitters.py", line 182, in sync_flush
    Emitter.flush(self)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/emitters.py", line 149, in flush
    self.send_events(self.buffer)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/snowplow_tracker/emitters.py", line 206, in send_events
    status_code = self.http_post(data).status_code
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/dbt/tracking.py", line 68, in http_post
    r = requests.post(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/connection.py", line 343, in connect
    self.ssl_context = create_urllib3_context(
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 290, in create_urllib3_context
    context.verify_mode = cert_reqs
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 438, in verify_mode
    self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/OpenSSL/SSL.py", line 1028, in set_verify
    self._verify_helper = _VerifyHelper(callback)
  File "/opt/homebrew/Cellar/dbt/0.19.1_1/libexec/lib/python3.8/site-packages/OpenSSL/SSL.py", line 331, in __init__
    self.callback = _ffi.callback(
MemoryError: Cannot allocate write+execute memory for ffi.callback(). You might be running on a system that prevents this. For more information, see https://cffi.readthedocs.io/en/latest/using.html#callbacks
```
This is a weird one. The error is in `pyOpenSSL` (https://github.com/pyca/pyopenssl/issues/873#issuecomment-778588689), and it's cropping up in anonymous usage tracking (`snowplow_tracker` &rarr; `requests` &rarr; `urllib3` &rarr; `pyOpenSSL`).

But the reason `pyOpenSSL` is installed in the first place is because it's explicitly required by `snowflake-connector-python`. If you're using a non-Snowflake adapter plugin, e.g. `dbt-postgres`, `pyOpenSSL` is never installed, `urllib3` falls back to using the standard-library `ssl` module, and everything is fine. (The standard `ssl` library can be insecure for python2, but it sounds like less of a concern on python3. See: [urllib3 docs](https://urllib3.readthedocs.io/en/latest/reference/contrib/pyopenssl.html), and for more explanation, section 1.4.1 of [this PDF](https://urllib3.readthedocs.io/_/downloads/en/1.23/pdf/).)

Since dbt only supports python3, I'd like to figure out a way to tell `urllib3` (required for anonymous usage tracking) to ignore `pyOpenSSL` if it's installed. It sounds like [there may be a way to do it](https://urllib3.readthedocs.io/en/latest/reference/contrib/pyopenssl.html?highlight=pyopenssl#urllib3.contrib.pyopenssl.extract_from_urllib3), via `urllib3.contrib.pyopenssl.extract_from_urllib3()`, but I'm not sure where we'd put that call, or if we're accessing it at too high a level.

@vklimontovich I'm glad you were able to get this working in the meantime by disabling anonymous usage tracking. Of course, I'd like to avoid it being an either/or choice!

cc @kwigley in case you find this interesting :)
@jtcohen6 wouldn't wrapping tracking code into `try/excerpt` a good temporary solution? At least it won't fail :(

Something like this one:

```
diff --git a/core/dbt/tracking.py b/core/dbt/tracking.py
index d950bb84..576e949c 100644
--- a/core/dbt/tracking.py
+++ b/core/dbt/tracking.py
@@ -425,7 +425,10 @@ def track_invalid_invocation(

 def flush():
     logger.debug("Flushing usage events")
-    tracker.flush()
+    try:
+        tracker.flush()
+    except:
+        logger.warn("Unable to flush usage events")


 def disable_tracking():
```
@vklimontovich Have you been able to confirm that the addition of that try/except results in error-free runs? (i.e. by cloning `dbt`, editing, and installing from local)

That's fine as a temporary solution—if you can confirm it works, I'd welcome a PR for it. In addition, I'd like to find a way to work around the `pyOpenSSL` bug, if possible.
@jtcohen6 Is this issue supposed to be resolved with `dbt 0.20.0`?  After upgrading, I am able to run `dbt --version` however `dbt run` is now giving me this error.  I tried the various work arounds but curious if there is an update? Thanks.
@brittianwarner We didn't include the proposed fix above in v0.20.0. You're welcome to try turning off anonymous tracking, and see if that fixes it.

Over the past few weeks, there have been a few reports of successful installs on M1s by other means: https://github.com/dbt-labs/dbt/issues/3239#issuecomment-868996761
@jtcohen6 do you have any insight into when the patch to resolve this issue will be released so we do not need to find these work arounds?
@kwigley and I just took a look at this, trying to figure out if there's a way we can undo [the urllib3/pyopenssl monkey patch](https://urllib3.readthedocs.io/en/latest/reference/contrib/pyopenssl.html#urllib3.contrib.pyopenssl.extract_from_urllib3) that `snowflake-connector-python` turns on [here](https://github.com/snowflakedb/snowflake-connector-python/blob/498e84c974a74072f899e6e5416a0317fee9d2f7/src/snowflake/connector/network.py#L104-L107).

If we can't figure that out—or even if we can—let's add `try/except` around `tracker.flush()` [here](https://github.com/dbt-labs/dbt/blob/26797921999d22c58491c0dfc1c81630f49789f7/core/dbt/tracking.py#L455-L457). As a general principle, a failure to send anonymous usage tracking shouldn't prevent folks from using dbt.

We'll do our best to sneak in this fix for v0.20.1.

Resolving patch:
diff --git a/core/dbt/tracking.py b/core/dbt/tracking.py
--- a/core/dbt/tracking.py
+++ b/core/dbt/tracking.py
@@ -454,7 +454,12 @@ def track_partial_parser(options):
 
 def flush():
     logger.debug("Flushing usage events")
-    tracker.flush()
+    try:
+        tracker.flush()
+    except Exception:
+        logger.debug(
+            "An error was encountered while trying to flush usage events"
+        )
 
 
 def disable_tracking():


Test patch: