eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am defining a python project using a pyproject.toml file. No setup.py, no setup.cfg. The project has dependencies on an alternate repository: https://artifactory.mypypy.com, how do I specify it ?

Answer: You put this in pyproject.toml:  [[tool.poetry.source]] name = "internal-repo-2" url = "https://<private-repo-2>" priority = "explicit"  There are alternative for priority but they come with a security risk: an attacker who learns the name of your internal package can push a package of the same name to PyPI, and it will then become part of your executable.
Question: Suppose I have a fully type hinted method with only keyword arguments : class A: def func(a: int, b: str, c: SomeIntricateTypeHint) -> OutputClass: ...  Now suppose I have a function that takes in variable keyword arguments, and passes them entirely onto that method : def outer_func(n_repeat: int, **kwargs: ???) -> OtherOutputClass: a = A() for _ in range(n_repeat): a.func(**kwargs)  In doing so, I have lost the benefits of the type hints for func. How do I type hint kwargs in outer_func such that I recover those benefits ? For extra detail, in my case, I don't personally define func. It's actually a method from a boto3 client object. As a result I'm looking for a solution that dynamically creates the type hint, rather than having to manually create the TypedDict.

Answer: In my case, it is json.loads wrapper. My get_json accept the bytes and doing the decoding but the json.loads will accept str | bytes | bytearray as first argument. from typing import ( ParamSpec, Callable, Generic, Concatenate, Any, ) P = ParamSpec("P") def __get_json_func_creator( _: Callable[ Concatenate[Any, P], Any ] = json.loads ): def func(data: bytes, *args: P.args, **kwargs: P.kwargs): return json.loads(__get_text(data), *args, **kwargs) return func; get_json = __get_json_func_creator()  In your case, you're defining func in class A. Assume that you're missing self argument in method declaration and are intended to strip a and b arguments? class A: def func( self, a: int, b: str, c: SomeIntricateTypeHint = None ) -> OutputClass: ... def __wrap_class_a_func( _: Callable[ Concatenate[A, Any, Any, P], Any ] = A.func ): def func(n_repeat: int, *args: P.args, **kwargs: P.kwargs): a = A() for num in range(n_repeat): a.func(num, str(num), **kwargs) return func outer_func = __wrap_class_a_func()  Unfortunately, in this case if you want to strip all positional parameter, you will have to define all positional type in Concatenate
Question: How to fix this error? C:\Users\vanvl\OneDrive\Bureaublad\Progammeren\Project 1.02.2>python Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from Custom_Widgets import ProjectMaker Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Users\vanvl\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\Custom_Widgets\ProjectMaker.py", line 14, in <module> import cairosvg File "C:\Users\vanvl\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\cairosvg\__init__.py", line 26, in <module> from . import surface # noqa isort:skip File "C:\Users\vanvl\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\cairosvg\surface.py", line 9, in <module> import cairocffi as cairo File "C:\Users\vanvl\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\cairocffi\__init__.py", line 48, in <module> cairo = dlopen( File "C:\Users\vanvl\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\cairocffi\__init__.py", line 45, in dlopen raise OSError(error_message) # pragma: no cover OSError: no library called "cairo-2" was found no library called "cairo" was found no library called "libcairo-2" was found cannot load library 'libcairo.so.2': error 0x7e cannot load library 'libcairo.2.dylib': error 0x7e cannot load library 'libcairo-2.dll': error 0x7e 

Answer: This is quite an annoying dependency problem because cairocffi is not built for windows, you need the additional dependency as explained there: https://doc.courtbouillon.org/cairocffi/stable/overview.html#installing-cairo-on-windows A quicker solution is to do the following: I used pipwin which install from an unofficial repository https://www.lfd.uci.edu/~gohlke/pythonlibs/#pycairo pip install pipwin pipwin install cairocffi  Also see this related issue: get cairosvg working in windows
Question: For some reason the string representation of negative imaginary numbers in Python is different for equal values: >>> str(-3j) '(-0-3j)' >>> str(0-3j) '-3j'  Moreover, if I try to get the string representation of -0-3j, I get: >>> str(-0-3j) '-3j'  Which seems like an inconsistency. The problem is that I use the string representation of complex scalar tensors to compute the hash of some operators, and because of this inconsistency I get different hashes for equal operators. Is there any way to convert '(-0-3j)' to '-3j'? Edit: Mechanic Pig pointed out that: >>> str(-0.-3j) '(-0-3j)' 

Answer: This issue is filed as a bug and stimulated appearance of PEP-0682 - Format Specifier for Signed Zero and originates from Floating-point arithmetic limitations. Once the bug is resolved -0 will not scare unnecessarily any more. But until then... You can adapt the snippet from this answer: def real_norm(no): return no if no != 0 else abs(no) def imag_norm(ino): return complex(real_norm(ino.real), real_norm(ino.imag)) test_cases = [-0-3j, 0-3j, -0-0j, 0-0j, -0+0j] for tc in test_cases: tn = imag_norm(tc) print(f"{tc} -> {tn}")
Question: I have an application that uses SQLAlchemy and Alembic for migrations. The repository looks like this: my-app/ my_app/ ... # Source code migrations/ versions/ ... # Migration scripts env.py alembic.ini MANIFEST.in README.rst setup.py  When in the repo, I can call alembic commands (alembic revision, alembic upgrade). I want to ship the app as a package to allow users to pip install, and I would like them to be able to just alembic upgrade head to migrate their DB. How can I achieve this? alembic is listed as a dependency. What I don't know is how to ensure alembic.ini and revision files are accessible to the alembic command without the user having to pull the repo. Adding them to MANIFEST.in will add them to the source package but AFAIU, when installing with pip, only my_app and subfolders end up in the (virtual) environment (this plus entry points). Note: the notions of source dist, wheel, MANIFEST.in and include_package_data are still a bit blurry to me but hopefully the description above makes the use case clear.

Answer: The obvious part of the answer is "include migration files in app directory". my-app/ my_app/ ... # Source code migrations/ versions/ ... # Migration scripts env.py alembic.ini MANIFEST.in README.rst setup.py  The not so obvious part is that when users install the package, they are not in the app directory, so they would need to specify the location of the alembic.ini file as a command line argument to use alembic commands (and this path is somewhere deep into a virtualenv). Not so nice. From a discussion with Alembic author, the recommended way is to provide user commands using the Alembic Python API internally to expose only a subset of user commands. Here's what I did. In migrations directory, I added this __init__.py file: """DB migrations""" from pathlib import Path from alembic.config import Config from alembic import command ROOT_PATH = Path(__file__).parent.parent ALEMBIC_CFG = Config(ROOT_PATH / "alembic.ini") def current(verbose=False): command.current(ALEMBIC_CFG, verbose=verbose) def upgrade(revision="head"): command.upgrade(ALEMBIC_CFG, revision) def downgrade(revision): command.downgrade(ALEMBIC_CFG, revision)  Then in a commands.py file in application root, I added a few commands: @click.command() @click.option("-v", "--verbose", is_flag=True, default=False, help="Verbose mode") def db_current_cmd(verbose): """Display current database revision""" migrations.current(verbose) @click.command() @click.option("-r", "--revision", default="head", help="Revision target") def db_upgrade_cmd(revision): """Upgrade to a later database revision""" migrations.upgrade(revision) @click.command() @click.option("-r", "--revision", required=True, help="Revision target") def db_downgrade_cmd(revision): """Revert to a previous database revision""" migrations.downgrade(revision)  And of course, in setup.py setup( ... entry_points={ "console_scripts": [ ... "db_current = my_app.commands:db_current_cmd", "db_upgrade = my_app.commands:db_upgrade_cmd", "db_downgrade = my_app.commands:db_downgrade_cmd", ], }, )
Question: I am on a workstation with 4 A6000 GPUs. Moving a Torch tensor from one GPU to another GPU corrupts the data, silently!!! See the simple example below. x >tensor([1], device='cuda:0') x.to(1) >tensor([1], device='cuda:1') x.to(2) >tensor([0], device='cuda:2') x.to(3) >tensor([0], device='cuda:3')  Any ideas what is the cause of this issue? Other info that might be handy: (there was two nvlinks which I manually removed trying to solve the problem)  GPU0 GPU1 GPU2 GPU3 CPU Affinity NUMA Affinity GPU0 X SYS SYS SYS 0-63 N/A GPU1 SYS X SYS SYS 0-63 N/A GPU2 SYS SYS X SYS 0-63 N/A GPU3 SYS SYS SYS X 0-63 N/A  nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Tue_Mar__8_18:18:20_PST_2022 Cuda compilation tools, release 11.6, V11.6.124 Build cuda_11.6.r11.6/compiler.31057947_0  NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6  Edit: adding some screenshots It seems to be stateful. Changes which GPUs work fine together after starting a new python runtime. 

Answer: The solution was to disable IOMMU. On our server, in the BIOS settings  /Advanced/AMD CBS/NBIO Common Options/IOMMU -> IOMMU - Disabled See the PyTorch issues thread for more information.
Question:  class User(Base): __tablename__ = 'users' id = Column(Integer, primary_key=True) age = Column(Integer) # I need age to have min value of 0 and max of 100 email = Column(String)  SQLAlchemy documentation says that there is no such attributes to pass while creating a column

Answer: As described in brunson's answer, you can add a check constraint to perform the validation in the database, if the database supports check constraints. import sqlalchemy as sa ... class Test(Base): __tablename__ = 'test' id = sa.Column(sa.Integer, primary_key=True) age = sa.Column(sa.Integer, sa.CheckConstraint('age > 0 AND age < 100'))  It may be more convenient to perform the validation before data is sent to the database, in the application layer. In this case, the orm.validates decorator can be used: class Test(Base): __tablename__ = 'test' id = sa.Column(sa.Integer, primary_key=True) age = sa.Column(sa.Integer, sa.CheckConstraint('age > 0 AND age < 100')) @orm.validates('age') def validate_age(self, key, value): if not 0 < value < 100: raise ValueError(f'Invalid age {value}') return value
Question: command: pipenv lock --requirements --keep-outdated output: Usage: pipenv lock [OPTIONS] Try 'pipenv lock -h' for help. Error: No such option: --requirements Did you mean --quiet?  Any idea how to fix this?

Answer: the -r option on pipenv lock command is deprecated for some time. use the requirements option to generate the requirements.txt ie: pipenv requirements > requirements.txt (Default dependencies) and to freeze dev dependencies as well use the --dev option pipenv requirements --dev > dev-requirements.txt  Sometimes, you would want to generate a requirements file based on your current environment, for example to include tooling that only supports requirements.txt. You can convert a Pipfile and Pipfile.lock into a requirements.txt file very easily.  see also: https://pipenv.pypa.io/en/latest/advanced/#generating-a-requirements-txt
Question: I wish to flatten(I am not sure whether is the correct thing to call it flatten) the columns with rows. multiple rows into single row with column change to column_rows I have a dataframe as below: data = {"a":[3,4,5,6], "b":[88,77,66,55], "c":["ts", "new", "thing", "here"], "d":[9.1,9.2,9.0,8.4]} df = pd.DataFrame(data)  my current output is:  a b c d 0 3 88 ts 9.1 1 4 77 new 9.2 2 5 66 thing 9.0 3 6 55 here 8.4  my expected otput:  a_0 a_1 a_2 a_3 b_0 b_1 b_2 b_3 c_0 c_1 c_2 c_3 d_0 d_1 d_2 d_3 0 3 4 5 6 88 77 66 55 ts new thing here 9.1 9.2 9.0 8.4  from shape (4,4) to (1, 16)

Answer: Update let's use the walrus operator new in Python 3.8 to create a one-liner: (df_new := df.unstack().to_frame().T).set_axis( [f"{i}_{j}" for i, j in df_new.columns], axis=1 )  Output:  a_0 a_1 a_2 a_3 b_0 b_1 b_2 b_3 c_0 c_1 c_2 c_3 d_0 d_1 d_2 d_3 0 3 4 5 6 88 77 66 55 ts new thing here 9.1 9.2 9.0 8.4   Try this, using unstack, to_frame and transpose. Next, flatten the column headers using list comprehension: df_new = df.unstack().to_frame().T df_new.columns = [f'{i}_{j}' for i, j in df_new.columns] df_new  Output:  a_0 a_1 a_2 a_3 b_0 b_1 b_2 b_3 c_0 c_1 c_2 c_3 d_0 d_1 d_2 d_3 0 3 4 5 6 88 77 66 55 ts new thing here 9.1 9.2 9.0 8.4
Question: I use MultiHeadAttention layer in my transformer model (my model is very similar to the named entity recognition models). Because my data comes with different lengths, I use padding and attention_mask parameter in MultiHeadAttention to mask padding. If I would use the Masking layer before MultiHeadAttention, will it have the same effect as attention_mask parameter? Or should I use both: attention_mask and Masking layer?

Answer: The Tensoflow documentation on Masking and padding with keras may be helpful. The following is an excerpt from the document.  When using the Functional API or the Sequential API, a mask generated by an Embedding or Masking layer will be propagated through the network for any layer that is capable of using them (for example, RNN layers). Keras will automatically fetch the mask corresponding to an input and pass it to any layer that knows how to use it.  tf.keras.layers.MultiHeadAttention also supports automatic mask propagation in TF2.10.0.  Improved masking support for tf.keras.layers.MultiHeadAttention.    Implicit masks for query, key and value inputs will automatically be used to compute a correct attention mask for the layer. These padding masks will be combined with any attention_mask passed in directly when calling the layer. This can be used with tf.keras.layers.Embedding with mask_zero=True to automatically infer a correct padding mask.     Added a use_causal_mask call time arugment to the layer. Passing use_causal_mask=True will compute a causal attention mask, and optionally combine it with any attention_mask passed in directly when calling the layer.
Question: Some python processes crash with: objc[51435]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.  Those are processes using child shells, forking threads, etc. MacOS blocks them for some security reasons (which I am not sure what are, but that is what people say) The solution is to disable this security check: export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES  Which is fine for known libraries and dependencies, and within the currently running shell. Is it safe to set it as a global environment variable, disabling this check globally in my local mac machine?

Answer: Apple changed the way fork() behaves in High Sierra (>= 10.13). If enabled, the OBJC_DISABLE_INITIALIZE_FORK_SAFETY variable turns off the immediate crash behaviour that their newer ObjectiveC framework otherwise enforces by default, as part of this change. Your question "is it safe to set it as a global environment variable" depends on your definition of "safe" in this context. It's safe in the sense your computer won't burst into flames. It's unsafe in the sense it may mask crash information that would be otherwise presented by an app that goes awry, and may allow a fork-bomb type process to crash your computer. So if you only have one use case where setting the flag is strictly necessary, then it's best to localise its setting to just that script/scenario.
Question: So I wanted to try some reinforcement learning, I haven't coded anything for a while. On Jupiter Notebooks when I run this code import gym env = gym.make("MountainCar-v0") env.reset() done = False while not done: action = 2 # always go right! env.step(action) env.render()  it just tries to render it but can't, the hourglass on top of the window is showing but it never renders anything, I can't do anything from there. Same with this code import gym env_name = "MountainCar-v0" env = gym.make(env_name) env.reset() for _ in range(200) action = env.action_space.sample() env.step(action) env.render()  Both of these don't work neither on Jupiter notebooks nor Pycharm nor terminal. I'm on Windows. Couldn't find anything similar to this online. Yes I'm new to this Edit- I did this # Install latest stable version from PyPI !pip install -U pysdl2 # Install latest development verion from GitHub !pip install -U git+https://github.com/py-sdl/py-sdl2.git  and it now says error: windlib not available I tried !pip install windlib  but still can't fix the error

Answer: Use an older version that supports your current version of Python. I solved the problem using gym 0.17.3 pip install gym==0.17.3  and the code: import gym env = gym.make("MountainCar-v0") state = env.reset() done = False while not done: action = 2 # always go right! env.step(action) print(new_state, render) env.render(mode="human") env.close()
Question: This knowledge post isn't a duplication of other similar ones, since it's related to 12/September/2022 Xcode update, which demands a different kind of solution I have come to my computer today and discovered that nothing runs on my terminal Every time I have opened my IDE (VS Code or PyCharm), it has given me this message in the start of the terminal. I saw so many solutions, which have said to uninstall pyenv and install python via brew, which was a terrible idea, because I need different python versions for different projects. Also, people spoke a lot about symlinks, which as well did not make any sense, because everything was working until yesterday. Furthermore, overwriting .oh-my-zsh with a new built one did not make any difference.

Answer: I was prompted to reinstall commandLine tools over and over when trying to accept the terms I FIXED this by opening xcode and confirming the new update information
Question: I have seen a lot of similar questions but none seem to work for my case. I'm pretty sure this is just a groupby transform but I keep getting KeyError along with axis issues. I am trying to groupby filename and check count where pred != gt. For example Index 2 is the only one for f1.wav so 1, and Index (13,14,18) for f2.wav so 3. df = pd.DataFrame([{'pred': 0, 'gt': 0, 'filename': 'f1.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f1.wav'}, {'pred': 2, 'gt': 0, 'filename': 'f1.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f1.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f1.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f1.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f1.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f1.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f1.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f1.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f2.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f2.wav'}, {'pred': 2, 'gt': 2, 'filename': 'f2.wav'}, {'pred': 0, 'gt': 2, 'filename': 'f2.wav'}, {'pred': 0, 'gt': 2, 'filename': 'f2.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f2.wav'}, {'pred': 0, 'gt': 0, 'filename': 'f2.wav'}, {'pred': 2, 'gt': 2, 'filename': 'f2.wav'}, {'pred': 0, 'gt': 2, 'filename': 'f2.wav'}, {'pred': 2, 'gt': 0, 'filename': 'f2.wav'}])   pred gt filename 0 0 0 f1.wav 1 0 0 f1.wav 2 2 0 f1.wav 3 0 0 f1.wav 4 0 0 f1.wav 5 0 0 f1.wav 6 0 0 f1.wav 7 0 0 f1.wav 8 0 0 f1.wav 9 0 0 f1.wav 10 0 0 f2.wav  Expected output  pred gt filename counts 0 0 0 f1.wav 1 1 0 0 f1.wav 1 2 2 0 f1.wav 1 3 0 0 f1.wav 1 4 0 0 f1.wav 1 5 0 0 f1.wav 1 6 0 0 f1.wav 1 7 0 0 f1.wav 1 8 0 0 f1.wav 1 9 0 0 f1.wav 1 10 0 0 f2.wav 3 11 0 0 f2.wav 3 12 2 2 f2.wav 3 13 0 2 f2.wav 3 14 0 2 f2.wav 3 15 0 0 f2.wav 3 16 0 0 f2.wav 3 17 2 2 f2.wav 3 18 0 2 f2.wav 3 19 2 0 f2.wav 3  I was thinking df.groupby('filename').transform(lambda x: x['pred'].ne(x['gt']).sum(), axis=1) but I get TypeError: Transform function invalid for data types

Answer: .transform operates on each column individually, so you won't be able to access both 'pred' and 'gt' in a transform operation. This leaves you with 2 options:  aggregate and reindex or join back to the original shape pre-compute the boolean array and .transform on that  approach 2 will probably be the fastest here: df['counts'] = ( (df['pred'] != df['gt']) .groupby(df['filename']).transform('sum') ) print(df) pred gt filename counts 0 0 0 f1.wav 1 1 0 0 f1.wav 1 2 2 0 f1.wav 1 3 0 0 f1.wav 1 4 0 0 f1.wav 1 5 0 0 f1.wav 1 6 0 0 f1.wav 1 7 0 0 f1.wav 1 8 0 0 f1.wav 1 9 0 0 f1.wav 1 10 0 0 f2.wav 4 11 0 0 f2.wav 4 12 2 2 f2.wav 4 13 0 2 f2.wav 4 14 0 2 f2.wav 4 15 0 0 f2.wav 4 16 0 0 f2.wav 4 17 2 2 f2.wav 4 18 0 2 f2.wav 4 19 2 0 f2.wav 4  Note that f2.wav has 4 instances where 'pre' != 'gt' (index 13, 14, 18, 19)
Question: in a simple test in pytorch, I want to see grad in a non-leaf tensor, so I use retain_grad(): import torch a = torch.tensor([1.], requires_grad=True) y = torch.zeros((10)) gt = torch.zeros((10)) y[0] = a y[1] = y[0] * 2 y.retain_grad() loss = torch.sum((y-gt) ** 2) loss.backward() print(y.grad)  it gives me a normal output: tensor([2., 4., 0., 0., 0., 0., 0., 0., 0., 0.])  but when I use retain grad() before y[1] and after y[0] is assigned: import torch a = torch.tensor([1.], requires_grad=True) y = torch.zeros((10)) gt = torch.zeros((10)) y[0] = a y.retain_grad() y[1] = y[0] * 2 loss = torch.sum((y-gt) ** 2) loss.backward() print(y.grad)  now the output changes to: tensor([10., 0., 0., 0., 0., 0., 0., 0., 0., 0.])  I can't understand the result at all.

Answer: Okay so what's going on is really weird. What .retain_grad() essentially does is convert any non-leaf tensor into a leaf tensor, such that it contains a .grad attribute (since by default, pytorch computes gradients to leaf tensors only). Hence, in your first example, after calling y.retain_grad(), it basically converted y into a leaf tensor with an accessible .grad attribute. However, in your second example, you initially converted the entire y tensor into a leaf tensor; then, you created a non-leaf tensor (y[1]) within your leaf tensor (y), which is what caused the confusion. y = torch.zeros((10)) # y is a non-leaf tensor y[0] = a # y[0] is a non-leaf tensor y.retain_grad() # y is a leaf tensor (including y[1]) y[1] = y[0] * 2 # y[1] is a non-leaf tensor, BUT y[0], y[2], y[3], ..., y[9] are all leaf tensors!  The confusing part is: y[1] after calling y.retain_grad() is now a leaf tensor with a .grad attribute. However, y[1] after the computation (y[1] = y[0] * 2) is now not a leaf tensor with a .grad attribute; it is now treated as a new non-leaf variable/tensor. Therefore, when calling loss.backward(), the Chain rule of the loss w.r.t y, and particularly looking at the Chain rule of the loss w.r.t leaf y[1] now looks something like this: