eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Is there a python native way to connect django to a database through an ssh tunnel? I have seen people using ssh port forwarding in the host machine but I would prefer a solution that can be easily containerized.

Answer: It is pretty seamless. Requirements: The sshtunnel package https://github.com/pahaz/sshtunnel  In the django settings.py create an ssh tunnel before the django DB settings block:  from sshtunnel import SSHTunnelForwarder # Connect to a server using the ssh keys. See the sshtunnel documentation for using password authentication ssh_tunnel = SSHTunnelForwarder( SERVER_IP, ssh_private_key=PATH_TO_SSH_PRIVATE_KEY, ssh_private_key_password=SSH_PRIVATE_KEY_PASSWORD, ssh_username=SSH_USERNAME, remote_bind_address=('localhost', LOCAL_DB_PORT_ON_THE_SERVER), ) ssh_tunnel.start()   Then add the DB info block in the settings.py. Here I am adding a default local DB and the remote DB that we connect to using the ssh tunnel  DATABASES = { 'default': { 'ENGINE': 'django.contrib.gis.db.backends.postgis', 'HOST': NORMAL_DB_HOST, 'PORT': NORMAL_DB_PORT, 'NAME': NORMAL_DB_NAME, 'USER': NORMAL_DB_USER, 'PASSWORD': NORMAL_DB_PASSWORD, }, 'shhtunnel_db': { 'ENGINE': 'django.contrib.gis.db.backends.postgis', 'HOST': 'localhost', 'PORT': ssh_tunnel.local_bind_port, 'NAME': REMOTE_DB_DB_NAME, 'USER': REMOTE_DB_USERNAME, 'PASSWORD': REMOTE_DB_PASSWORD, }, }  That is it. Now one can make migratations to the remote db using commands like $ python manage.py migrate --database=shhtunnel_db or make calls to the db from within the python code using lines like Models.objects.all().using('shhtunnel_db') Extra: In my case the remote db was created by someone else and I only wanted to read it. In order to avoid writing the models and deactivating the model manager I used the following django command to get the models from the database [src]: python manage.py inspectdb
Question: I have sqlalchemy-stubs installed via my Pipfile: [dev-packages] sqlalchemy-stubs = {editable = true, git = "https://github.com/dropbox/sqlalchemy-stubs.git"}  Verified by running pipenv graph: sqlalchemy-stubs==0.4 - mypy [required: >=0.790, installed: 0.910] - mypy-extensions [required: >=0.4.3,<0.5.0, installed: 0.4.3] - toml [required: Any, installed: 0.10.2] - typing-extensions [required: >=3.7.4, installed: 4.0.0] - typing-extensions [required: >=3.7.4, installed: 4.0.0]  However when running mypy (mypy --strict .) with the config: [mypy] plugins = graphene_plugin, sqlmypy  I get the error: error: Error importing plugin "sqlmypy": No module named 'sqlmypy'  What is going on?

Answer: I fixed by removing the editable = true from the pipfile.
Question: I am writing tests with pytest in pycharm. The tests are divided into various classes.  I would like to specify certain classes that have to run before other classes.  I have seen various questions on stackoverflow (such as specifying pytest tests to run from a file and how to run a method before all other tests).  These and various others questions wanted to choose specific functions to run in order. This can be done, I understand, using fixtures or with pytest ordering. I don't care which functions from each class run first. All I care about is that the classes run in the order I specify. Is this possible?

Answer: Approach You can use the pytest_collection_modifyitems hook to modify the order of collected tests (items) in place. This has the additional benefit of not having to install any third party libraries. With some custom logic, this allows to sort by class. Full example Say we have three test classes:  TestExtract TestTransform TestLoad  Say also that, by default, the test order of execution would be alphabetical, i.e.: TestExtract -> TestLoad -> TestTransform which does not work for us due to test class interdependencies. We can add pytest_collection_modifyitems to conftest.py as follows to enforce our desired execution order: # conftest.py def pytest_collection_modifyitems(items): """Modifies test items in place to ensure test classes run in a given order.""" CLASS_ORDER = ["TestExtract", "TestTransform", "TestLoad"] class_mapping = {item: item.cls.__name__ for item in items} sorted_items = items.copy() # Iteratively move tests of each class to the end of the test queue for class_ in CLASS_ORDER: sorted_items = [it for it in sorted_items if class_mapping[it] != class_] + [ it for it in sorted_items if class_mapping[it] == class_ ] items[:] = sorted_items  Some comments on the implementation details:  Test classes can live in different modules CLASS_ORDER does not have to be exhaustive. You can reorder just those classes on which to want to enforce an order (but note: if reordered, any non-reordered class will execute before any reordered class) The test order within the classes is kept unchanged It is assumed that test classes have unique names items must be modified in place, hence the final items[:] assignment
Question: I have that PyCharm is halting on all my exceptions, even the ones I am handling in a try except block. I do not want it to break there - I am handling and perhaps expecting an error. But every other exception I do want it to halt and suspend execution (e.g. so that I have the program state and debug it). How does one do that? I tried going into the python exception breakpoint option but I didn't see an option like "break only on unhandled exceptions" e.g as suggested by these:  Stop PyCharm If Error https://intellij-support.jetbrains.com/hc/en-us/community/posts/206601165-How-to-enable-stopping-on-unhandled-exceptions-  note this is my current state, note how it stopped in my try block... :( crossposted: https://intellij-support.jetbrains.com/hc/en-us/community/posts/4415666598546-How-to-stop-PyCharm-s-break-stop-halt-feature-on-handled-exceptions-i-e-only-break-on-python-unhandled-exceptions-   I tried: In your link here intellij-support.jetbrains.com/hc/en-us/community/posts/… the poster Okke said they solved this issue adding --pdb to the 'addition arguments', which someone later said they probably meant interpreter options.  but didn't work got error: /Users/brandomiranda/opt/anaconda3/envs/meta_learning/bin/python --pdb /Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py --cmd-line --multiproc --qt-support=auto --client 127.0.0.1 --port 58378 --file /Users/brandomiranda/ultimate-utils/tutorials_for_myself/try_catch_pycharm_issues/try_catch_with_pickle.py unknown option --pdb usage: /Users/brandomiranda/opt/anaconda3/envs/meta_learning/bin/python [option] ... [-c cmd | -m mod | file | -] [arg] ... Try `python -h' for more information. Process finished with exit code 2 

Answer: I think it is already working actually, but you are in fact not catching the correct error. In your code you have: try: pickle.dumps(obj) except pickle.PicklingError: return False  But the error thrown is AttributeError. So to avoid that you need something like this: try: pickle.dumps(obj) except (pickle.PicklingError, AttributeError): return False
Question: I have been using the python module nudenet for my final degree project. I'm using google colab to run it. It worked correctly and without any problem during this last months until yesterday, when I tried to import it, this error happenend: !pip install --upgrade nudenet from nudenet import NudeClassifier ImportError: cannot import name '_registerMatType' from 'cv2.cv2' (/usr/local/lib/python3.7/dist-packages/cv2/cv2.cpython-37m-x86_64-linux-gnu.so)  I tried to solve this error by downgrading opencv-python-headless to a previous version !pip uninstall opencv-python-headless==4.5.5.62 !pip install opencv-python-headless==4.5.1.48  But then, when I load the classifier this error appears: classifier = NudeClassifier() Downloading the checkpoint to /root/.NudeNet/classifier_model.onnx MB| |# | 0 Elapsed Time: 0:00:00 Content-length not found, file size cannot be estimated. Succefully Downloaded to: /root/.NudeNet/classifier_model.onnx InvalidProtobuf: [ONNXRuntimeError] : 7 : INVALID_PROTOBUF : Load model from /root/.NudeNet/classifier_model.onnx failed:Protobuf parsing failed.  I have also tried to downgrade the version of the module nudenet, and still nothing works. Thank you in advance. This is the link for the module in github 

Answer: replace the rile classifier_model.onnx with nudenet classifier and file classifier_lite.onnx with nudenet lite classifier
Question: Today I stumbled upon the following behaviour: class myobject(object): """Should behave the same as object, right?""" obj = myobject() obj.a = 2 # <- works obj = object() obj.a = 2 # AttributeError: 'object' object has no attribute 'a'  I want to know what is the logic behind designing the language to behave this way, because it feels utterly paradoxical to me. It breaks my intuition that if I create a subclass, without modification, it should behave the same as the parent class.  EDIT: A lot of the answers suggest that this is because we want to be able to write classes that work with __slots__ instead of __dict__ for performance reasons. However, we can do: class myobject_with_slots(myobject): __slots__ = ("x",) obj = myobject_with_slots() obj.x = 2 obj.a = 2 assert "a" in obj.__dict__ # ✔ assert "x" not in obj.__dict__ # ✔  So it seems we can have both __slots__ and __dict__ at the same time, so why doesn't object allow both, but one-to-one subclasses do?

Answer: Because derived classes do not necessarily support setattr either. class myobject(object): """Should behave the same as object!""" __slots__ = () obj = myobject() obj.a = 2 # <- works the same as for object  Since all types derive from object, most builtin types such as list are also examples. Arbitrary attribute assignment is something that object subclasses may support, but not all do. Thus, the common base class does not support this either.  Support for arbitrary attributes is commonly backed by the so-called __dict__ slot. This is a fixed attribute that contains a literal dict 1 to store any attribute-value pairs. In fact, one can manually define the __dict__ slot to get arbitrary attribute support. class myobject(object): """Should behave the same as object, right?""" __slots__ = ("__dict__",) obj = myobject() obj.a = 2 # <- works! print(obj.__dict__) # {'a': 2}  The takeaway from this demonstration is that fixed attributes is actually the "base behaviour" of Python; the arbitrary attributes support is built on top when required. Adding arbitrary attributes for object subtypes by default provides a simpler programming experience. However, still supporting fixed attributes for object subtypes allows for better memory usage and performance.  Data Model: __slots__ The space saved [by __slots__] over using __dict__ can be significant. Attribute lookup speed can be significantly improved as well.  Note that it is possible to define classes with both fixed attributes and arbitrary attributes. The fixed attributes will benefit from the improved memory layout and performance; since they are not stored in the __dict__, its memory overhead2 is lower – but it still costs.  1Python implementations may use different, optimised types for __dict__ as long as they behave like a dict. 2For its hash-based lookup to work efficiently with few collisions, a dict must be larger than the number of items it stores.
Question: I used to be able to collapse (fold) python dictionaries just fine in my VS Code. Randomly I am not able to do that anymore. I can still fold classes and functions just fine, but dictionaries cannot fold, the arrow on the left hand side just isn't there. I've checked my settings but I can't figure out what would've changed. I'm not sure the best forum to go to for help, so I'm hoping this is ok. Any ideas?

Answer: It's caused by Pylance v2022.1.1. Use v2022.1.0 instead. Issue #2248
Question: The following line pd.DataFrame({"col1": [1.1, 2.2]}, index=[3.3, 4.4])[2.5:3.5]  raises a mypy linting error of on the [2.5  Slice index must be an integer or None  This is valid syntax and correctly returns  col1 3.3 1.1  Without # type: ignore, how can I resolve this linting error? versions:  pandas 1.3.0 mypy 0.931  The code in question: def get_dataframe( ts_data: GroupTs, ts_group_name: str, start_time: Optional[float] = None, end_time: Optional[float] = None, ) -> pd.DataFrame: df = pd.DataFrame(ts_data.group[ts_group_name].ts_dict)[ start_time:end_time ].interpolate( method="index", limit_area="inside" ) # type: pd.DataFrame return df[~df.index.duplicated()] 

Answer: This is by design for now, I fear, but if you have to, you can silence mypy by slicing with a callable, like this: import pandas as pd df = pd.DataFrame({"col1": [1.1, 2.2]}, index=[3.3, 4.4])[ lambda x: (2.5 <= x.index) & (x.index < 3.5) ] print(df) # Ouput col1 3.3 1.1  And so mypy reports no issues found on this code.
Question: After working through the tutorial of SQLModel, I don't remember seeing anything on how to implement 1:1 relationships using Relationship attributes. I found documentation for SQLAlchemy, but it's not immediately clear how this applies to SQLModel. Code example: How to enforce that User and ICloudAccount have a 1:1 relationship? class User(SQLModel, table=True): id: Optional[int] = Field(default=None, primary_key=True) name: str icloud_account_id: Optional[int] = Field(default=None, foreign_key="icloudaccount.id") icloud_account: Optional["ICloudAccount"] = Relationship(back_populates="users") class ICloudAccount(SQLModel, table=True): id: Optional[int] = Field(default=None, primary_key=True) user_name: str users: List[User] = Relationship(back_populates="icloud_account") 

Answer: You can turn off the list functionality to allow SQLModel to foreign key as a one-to-one. You do this with the SQLalchemy keyword uselist class User(SQLModel, table=True): id: Optional[int] = Field(default=None, primary_key=True) name: str icloud_account_id: Optional[int] = Field(default=None, foreign_key="icloudaccount.id") icloud_account: Optional["ICloudAccount"] = Relationship(back_populates="user") class ICloudAccount(SQLModel, table=True): id: Optional[int] = Field(default=None, primary_key=True) user_name: str user: Optional["User"] = Relationship( sa_relationship_kwargs={'uselist': False}, back_populates="icloud_account" )
Question: I have a project and an existing virtual environment created with poetry (poetry install/init). So, as far as I know, the purpouse of a virtual environment is avoiding to modify the system base environment and the possibility of isolation (per project, per development, per system etc...). How can I create another brand new environment for my project in poetry? How can I eventually duplicate and use an existing one? I mean that the current one (activated) should be not involved in this (except for eventually copying it) because I want to test another set of dependencies and code. I am aware of this:  https://github.com/python-poetry/poetry/issues/4055 (answer is not clear and ticket is not closed) https://python-poetry.org/docs/managing-environments/ (use command seems not to work in the requested way) 

Answer: Poetry seems to be bound to one virtualenv per python interpreter. Poetry is also bound to the pyproject.toml file and its path to generate a new environment. So there are 2 tricky solutions: 1 - change your deps in the pyproject.toml and use another python version (installed for example with pyenv) and then: poetry env use X.Y  poetry will create a new virtual environment but this is not exactly the same as changing just some project deps. 2 - use another pyproject.toml from another path: mkdir env_test cp pyproject.toml env_test/pyproject.toml cd env_test nano pyproject.toml # edit your dependencies poetry install # creates a brand new virtual environment poetry shell # run your script with the new environment  This will generate a new environment with just the asked dependencies changed. Both environments can be used at the same time. After the test, it is eventually possible to delete the new environment with the env command.
Question: I am trying to pass each element of foo_list into a function expensive_call, and get a list of all the items whose output of expensive_call is Truthy. I am trying to do it with list comprehensions, is it possible? Something like: Something like this:  result_list = [y := expensive_call(x) for x in foo_list if y]  or....  result_list = [y for x in foo_list if y := expensive_call(x)]  Note: This is not a solution because it call expensive call twice:  result_list = [expensive_call(x) for x in foo_list if expensive_call(x)]  And before someone recommends none list comprehension, I know one can do: result_list = [] for x in foo_list: result = expensive_call(x) result and result_list.append(result) 

Answer: Quoting from above:  result_list = [y for x in foo_list if y := expensive_call(x)]  It should work almost exactly as how you have it; just remember to parenthesize the assignment with the := operator, as shown below. foo_list = [1, 2, 3] def check(x): return x if x != 2 else None result_list = [y for x in foo_list if (y := check(x))] print(result_list)  Result: [1, 3]
Question: I'm building a docker image on cloud server via the following docker file: # base image FROM python:3 # add python file to working directory ADD ./ / # install and cache dependencies RUN pip install --upgrade pip RUN pip install RUST RUN pip install transformers RUN pip install torch RUN pip install slack_sdk RUN pip install slack_bolt RUN pip install pandas RUN pip install gensim RUN pip install nltk RUN pip install psycopg2 RUN pip install openpyxl ......  When installing the transformers package, the following error occurs: STEP 5: RUN pip install transformers Collecting transformers Downloading transformers-4.15.0-py3-none-any.whl (3.4 MB) Collecting filelock ...... Downloading click-8.0.3-py3-none-any.whl (97 kB) Building wheels for collected packages: tokenizers Building wheel for tokenizers (pyproject.toml): started ERROR: Command errored out with exit status 1: command: /usr/local/bin/python /usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmp_3y7hw5q cwd: /tmp/pip-install-bsy5f4da/tokenizers_e09b9f903acd40f0af4a997fe1d8fdb4 Complete output (50 lines): running bdist_wheel ...... copying py_src/tokenizers/trainers/__init__.pyi -> build/lib.linux-x86_64-3.10/tokenizers/trainers copying py_src/tokenizers/tools/visualizer-styles.css -> build/lib.linux-x86_64-3.10/tokenizers/tools running build_ext error: can't find Rust compiler If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler. To update pip, run: pip install --upgrade pip and then retry package installation. If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain. ---------------------------------------- ERROR: Failed building wheel for tokenizers ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects Building wheel for tokenizers (pyproject.toml): finished with status 'error' Failed to build tokenizers subprocess exited with status 1 subprocess exited with status 1 error building at STEP "RUN pip install transformers": exit status 1 time="2022-01-18T07:24:56Z" level=error msg="exit status 1" Dockerfile build failed - exit status 1exit status 1  I'm not very sure about what's happening here. Can anyone help me? Thanks in advance.

Answer: The logs say error: can't find Rust compiler  You need to install a rust compiler. See https://www.rust-lang.org/tools/install. You can modify the installation instructions for a docker image like this (from https://stackoverflow.com/a/58169817/5666087): RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y ENV PATH="/root/.cargo/bin:${PATH}"
Question: Having an instance c of a class C, I would like to make c immutable, but other instances of C dont have to. Is there an easy way to achieve this in python?

Answer: You can't make Python classes fully immutable. You can however imitate it: class C: _immutable = False def __setattr__(self, name, value): if self._immutable: raise TypeError(f"Can't set attribute, {self!r} is immutable.") super().__setattr__(name, value)  Example: >>> c = C() >>> c.hello = 123 >>> c.hello 123 >>> c._immutable = True >>> c.hello = 456 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 5, in __setattr__ TypeError: Can't set attribute, <__main__.C object at 0x000002087C679D20> is immutable.   If you wish to set it at initialization, you can add an __init__ like so: class C: _immutable = False def __init__(self, immutable=False): self._immutable = immutable def __setattr__(self, name, value): if self._immutable: raise TypeError(f"Can't set attribute, {self!r} is immutable.") super().__setattr__(name, value)   Keep in mind you can still bypass it by accessing and modifying the __dict__ of the instance directly: >>> c = C(immutable=True) >>> c.__dict__["hello"] = 123 >>> c.hello 123  You may attempt to block it like so: class C: _immutable = False def __init__(self, immutable=False): self._immutable = immutable def __getattribute__(self, name): if name == "__dict__": raise TypeError("Can't access class dict.") return super().__getattribute__(name) def __setattr__(self, name, value): if self._immutable: raise TypeError(f"Can't set attribute, {self!r} is immutable.") super().__setattr__(name, value)  But even then it's possible to bypass: >>> c = C(immutable=True) >>> object.__getattribute__(c, "__dict__")["hello"] = 123 >>> c.hello 123
Question: I am experimenting with polars and would like to understand why using polars is slower than using pandas on a particular example: import pandas as pd import polars as pl n=10_000_000 df1 = pd.DataFrame(range(n), columns=['a']) df2 = pd.DataFrame(range(n), columns=['b']) df1p = pl.from_pandas(df1.reset_index()) df2p = pl.from_pandas(df2.reset_index()) # takes ~60 ms df1.join(df2) # takes ~950 ms df1p.join(df2p, on='index') 

Answer: A pandas join uses the indexes, which are cached. A comparison where they do the same: # pandas # CPU times: user 1.64 s, sys: 867 ms, total: 2.5 s # Wall time: 2.52 s df1.merge(df2, left_on="a", right_on="b") # polars # CPU times: user 5.59 s, sys: 199 ms, total: 5.79 s # Wall time: 780 ms df1p.join(df2p, left_on="a", right_on="b")
Question: I have environment variables that I need to get from two different files in order to keep user+pw outside of the git repo. I download the sensitive user+pass from another location and add it to .gitignore. I am using from os import getenv from dotenv import load_dotenv ... load_dotenv() DB_HOST=getenv('DB_HOST') # from env file 1 DB_NAME=getenv('DB_NAME') # from env file 1 DB_USER=getenv('DB_USER') # from env file 2 DB_PASS=getenv('DB_PASS') # from env file 2  and I have the two ".env" files in the folder of the python script. env_file.env contains: DB_HOST=xyz DB_NAME=abc  env_file_in_gitignore.env which needs to stay out of the git repo but is available by download using an sh script: DB_USER=me DB_PASS=eao  How to avoid the error: TypeError: connect() argument 2 must be str, not None connect() argument 2 must be str, not None  which is thrown since one of the two files are not used for the .env import? How can I get environment variables from two different ".env" files, both stored in the working directory?

Answer: You can add file path as an argument in load_dotenv function from dotenv import load_dotenv import os load_dotenv(<file 1 path>) load_dotenv(<file 2 path>)