eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I've read all the documentations I could find about torch.distributed.barrier(), but still having trouble understanding how it's being used in this script and would really appreciate some help. So the official doc of torch.distributed.barrier says it "Synchronizes all processes.This collective blocks processes until the whole group enters this function, if async_op is False, or if async work handle is called on wait()." It's used in two places in the script:  First place  if args.local_rank not in [-1, 0] and not evaluate: torch.distributed.barrier() # Make sure only the first process in distributed training process the dataset, and the others will use the cache ... (preprocesses the data and save the preprocessed data) if args.local_rank == 0 and not evaluate: torch.distributed.barrier()  Second place  if args.local_rank not in [-1, 0]: torch.distributed.barrier() # Make sure only the first process in distributed training will download model & vocab ... (loads the model and the vocabulary) if args.local_rank == 0: torch.distributed.barrier() # Make sure only the first process in distributed training will download model & vocab  I'm having trouble relating the comment in the code to the functionality of this function stated in the official doc. How does it make sure only the first process executes the code between the two calls of torch.distributed.barrier() and why it only checks whether the local rank is 0 before the second call?  Thanks in advance!

Answer: First you need to understand the ranks. To be brief: in a multiprocessing context we typically assume that rank 0 is the first process or base process. The other processes are then ranked differently, e.g. 1, 2, 3, totalling four processes in total. Some operations are not necessary to be done in parallel or you just need one process to do some preprocessing or caching so that the other processes can use that data. In your example, if the first if statement is entered by the non-base processes (rank 1, 2, 3), they will block (or "wait") because they run into the barrier. They wait there, because barrier() blocks until all processes have reached a barrier, but the base process has not reached a barrier yet. So at this point the non-base processes (1, 2, 3) are blocked, but the base process (0) continues. The base process will do some operations (preprocess and cache data, in this case) until it reaches the second if-statement. There, the base process will run into a barrier. At this point, all processes have stopped at a barrier, meaning that all current barriers can be lifted and all processes can continue. Because the base process prepared the data, the other processes can now use that data. Perhaps the most important thing to understand is:  when a process encounters a barrier it will block the position of the barrier is not important (not all processes have to enter the same if-statement, for instance) a process is blocked by a barrier until all processes have encountered a barrier, upon which those barriers are lifted for all processes
Question: In the interactive window in vscode you press shift-enter to run the code you just typed and enter to go to the next line. Can I swap this? 

Answer: The current answers explain how to make "enter" execute the command but they don't give you the other half: make shift+enter insert a new line. Here is the complete solution (I aggregated multiple solutions from various answers to similar questions to credit to everyone else who helped). Bonus: I come from MATLAB so I also made "F9" execute selection in the ipython window. Open keybindings.json (ctrl+shift+p, open keyboard shortcuts (JSON) Add the following:  [ { "key": "enter", "command": "interactive.execute", "when": "!suggestWidgetVisible && resourceScheme == 'vscode-interactive'" }, { "key": "shift+enter", "command": "", "when": "!suggestWidgetVisible && resourceScheme == 'vscode-interactive'" }, { "key": "f9", "command": "jupyter.execSelectionInteractive", "when": "editorTextFocus && !findInputFocussed && !replaceInputFocussed && editorLangId == 'python'" } ]
Question: I am trying to run my script but keep getting this error: File ".\checkmypass.py", line 1, in <module> import requests line 3, in <module> response = requests.get(url) AttributeError: partially initialized module 'requests' has no attribute 'get' (most likely due to a circular import)  How can I fix it?

Answer: Make sure the name of the file is not the same as the module you are importing – this will make Python think there is a circular dependency. Also check the URL and the package you are using. "Most likely due to a circular import" refers to a file (module) which has a dependency on something else and is trying to be imported while it's already been imported. Once it's correct, you should have something like this: import requests r = requests.get("http://google.com") print(r.status_code) # 200
Question: I am developing with Python and commonly running code in an integrated terminal with Shift + Enter. However, when debugging the process seems to be more complicated. I need to copy the code, move focus to debug REPL (Ctrl + Shift + Y), paste, run and move focus back to the editor. Is there any easier way to do this?

Answer: If you use the vscode's integrated debugging you can set a shortcut for sending selection to debug Repl. I use this on my keybindings.json config file: { "key": "shift+alt+d", "command": "editor.debug.action.selectionToRepl" }  The difference from the "workbench.action.terminal.runSelectedText" command is that you actually have to select the line to send to the debug Repl, does not work for just putting the cursor on the line and hitting the shortcut.
Question: I have created a basic scatter plot to compare two variables using altair. I expect the variables to be strongly correlated and the points should end up on or close to the line of identity. How can I add the line of identity to the plot? I would like it to be a line similar to those created by mark_rule, but extending diagonally instead of vertically or horizontally. Here is as far as I have gotten: import altair as alt import numpy as np import pandas as pd norm = np.random.multivariate_normal([0, 0], [[2, 1.8],[1.8, 2]], 100) df = pd.DataFrame(norm, columns=['var1', 'var2']) chart = alt.Chart(df, width=500, height=500).mark_circle(size=100).encode( alt.X('var1'), alt.Y('var2'), ).interactive() line = alt.Chart( pd.DataFrame({'var1': [-4, 4], 'var2': [-4, 4]})).mark_line().encode( alt.X('var1'), alt.Y('var2'), ).interactive() chart + line  The problems with this example is that the line doesn't extend forever when zooming (like a rule mark) and that the plot gets automatically scaled to the line endings instead of only the points.

Answer: It's not perfect but you could make the line longer and set the scale domain. import altair as alt import numpy as np import pandas as pd norm = np.random.multivariate_normal([0, 0], [[2, 1.8],[1.8, 2]], 100) df = pd.DataFrame(norm, columns=['var1', 'var2']) chart = alt.Chart(df, width=500, height=500).mark_circle(size=100).encode( alt.X('var1', scale=alt.Scale(domain=[-4,4])), alt.Y('var2', scale=alt.Scale(domain=[-4,4])), ).interactive() line = alt.Chart( pd.DataFrame({'var1': [-100, 100], 'var2': [-100, 100]})).mark_line().encode( alt.X('var1'), alt.Y('var2'), ).interactive() chart + line
Question: I have a directory structure like the following in my serverless application(simplest app to avoid clutter) which I created using AWS SAM with Python 3.8 as the runtime: ├── common │ └── a.py ├── hello_world │ ├── __init__.py │ ├── app.py │ └── requirements.txt └── template.yaml  I would like to import common/a.py module inside the Lambda handler - hello_world/app.py. Now I know that I can import it normally in Python by adding the path to PYTHONPATH or sys.path, but it doesn't work when the code is run in Lambda inside a Docker container. When invoked, the Lambda handler function is run inside /var/task directory and the folder structure is not regarded.  I tried inserting /var/task/common, /var/common, /var and even /common to sys.path programmatically like this: import sys sys.path.insert(0, '/var/task/common') from common.a import Alpha  but I still get the error: ModuleNotFoundError: No module named 'common' I am aware of Lambda Layers but given my scenario, I would like to directly reference the common code in multiple Lambda functions without the need of uploading to a layer. I want to have something like the serverless-python-requirements plugin in the Serverless framework but in AWS SAM. So my question is, how should I add this path to common to PYTHONPATH or sys.path? Or is there an alternative workaround for this like [serverless-python-requirements][3] to directly import a module in a parent folder without using Lambda Layers?

Answer: I didn't find what I was looking for but I ended up with a solution to create a single Lambda function in the root which handles all the different API calls within the function. Yes my Lambda function is integrated with API Gateway, and I can get the API method and API path using event["httpMethod"] and event ["httpPath"] respectively. I can then put all the packages under the root and import them between each other. For example, say I have 2 API paths /items and /employees that need to be handled and both of them need to handle both GET and POST methods, the following code suffices: if event["path"] == '/items': if event["httpMethod"] == 'GET': ... elif event["httpMethod"] == 'POST': ... elif event["path"] == '/employees': if event["httpMethod"] == 'GET': ... if event["httpMethod"] == 'POST': ...  So now I can have as much packages under this Lambda function. For example, the following is how repository looks like now: ├── application │ └── *.py ├── persistence │ └── *.py ├── models │ └── *.py └── rootLambdaFunction.py └── requirements.txt  This way, I can import packages at will wherever I want within the given structure.
Question: Whenever I try to install TensorFlow with pip on Python 3.8, I get the error that TensorFlow is not found. I have realized later on that it is not supported by Python 3.8. How can I install TensorFlow on Python 3.8?

Answer: As of May 7, 2020, according to Tensorflow's Installation page with pip, Python 3.8 is now supported. Python 3.8 support requires TensorFlow 2.2 or later. You should be able to install it normally via pip.  Prior to May 2020: As you mentioned, it is currently not supported by Python 3.8, but is by Python 3.7. You want to have virtualenv installed. You also need Python 3.7. Then you can just start a virtualenv with -p python3.7 and install it using pip like you did before: virtualenv --system-site-packages -p python3.7 DEST_DIR source ./DEST_DIR/bin/activate pip install --upgrade pip pip install --upgrade tensorflow
Question: I installed yapf using: conda install yapf  and add next lines in my .vscode/settings.json file: { //"python.linting.pylintEnabled": true, //"python.linting.pycodestyleEnabled": false, //"python.linting.flake8Enabled": true, "python.formatting.provider": "yapf", "python.formatting.yapfArgs": [ " — style", "{based_on_style: pep8, indent_width: 4}" ], "python.linting.enabled": true, }  But I can't understand how to use it - it doesn't show any error in a bad-formatted script: import pandas as pd class MyClass(object): def __init__(self, some_value: int): self.value = some_value def one_more_function(self, another_value): print(another_value) myObject = MyClass(45) myObject.one_more_function(2) my__object2 = MyClass(324) print('ok') def some_foo(): """ """ pass 

Answer: The problem was in wrong settings. To use yapf, black or autopep8 you need:  Install yapf / black / autopep8 (pip install black) Configure .vscode/settings.json in the next way:  part of the file: { "python.linting.enabled": true, "python.linting.pylintPath": "pylint", "editor.formatOnSave": true, "python.formatting.provider": "yapf", // or "black" here "python.linting.pylintEnabled": true, }  Key option - "editor.formatOnSave": true, this mean yapf formats your document every time you save it.
Question: Looks like I have broken my python installation when I wanted to switch to python 3.8. Using Ubuntu 18.04. Trying to use the gi, gives the following error: $ python Python 3.8.1 (default, Dec 31 2019, 18:42:42) [GCC 7.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from gi.repository import GLib, Gio Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python3/dist-packages/gi/__init__.py", line 42, in <module> from . import _gi ImportError: cannot import name '_gi' from partially initialized module 'gi' (most likely due to a circular import) (/usr/lib/python3/dist-packages/gi/__init__.py)  Tried running update-alternatives for python, but it tells me there is only one python alternative configured (3.8). Tried to reinstall python3-gi and python3.8. Still the same problem

Answer: I had the same issue. I linked python3 to python3.6, for me it was pointing to 3.8. That solved the issue. cd /usr/bin/ rm python3 ln -s python3.6 python3  Thats all. Now my system started working fine.
Question: I just installed the latest version of Tensorflow via pip install tensorflow and whenever I run a program, I get the log message:  W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found  Is this bad? How do I fix the error?

Answer: Tensorflow 2.1+ What's going on? With the new Tensorflow 2.1 release, the default tensorflow pip package contains both CPU and GPU versions of TF. In previous TF versions, not finding the CUDA libraries would emit an error and raise an exception, while now the library dynamically searches for the correct CUDA version and, if it doesn't find it, emits the warning (The W in the beginning stands for warnings, errors have an E (or F for fatal errors) and falls back to CPU-only mode. In fact, this is also written in the log as an info message right after the warning (do note that if you have a higher minimum log level that the default, you might not see info messages). The full log is (emphasis mine):  2020-01-20 12:27:44.554767: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found 2020-01-20 12:27:44.554964: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.  Should I worry? How do I fix it? If you don't have a CUDA-enabled GPU on your machine, or if you don't care about not having GPU acceleration, no need to worry. If, on the other hand, you installed tensorflow and wanted GPU acceleration, check your CUDA installation (TF 2.1 requires CUDA 10.1, not 10.2 or 10.0). If you just want to get rid of the warning, you can adapt TF's logging level to suppress warnings, but that might be overkill, as it will silence all warnings.  Tensorflow 1.X or 2.0: Your CUDA setup is broken, ensure you have the correct version installed.
Question: I want to use nox in my project managed with poetry. What is not going well is that installing dev dependency in nox session. I have the noxfile.py as shown below: import nox from nox.sessions import Session from pathlib import Path __dir__ = Path(__file__).parent.absolute() @nox.session(python=PYTHON) def test(session: Session): session.install(str(__dir__)) # I want to use dev dependency here session.run("pytest")  How can I install dev dependency in nox session?

Answer: Currently, session.install doesn't support poetry and install just runs pip in the shell. You can activate poetry with a more general method session.run. Example: @nox.session(python=False) def tests(session): session.run('poetry', 'shell') session.run('poetry', 'install') session.run('pytest')  When you set up session, you can do everything by your own disabling creation of python virtualenv (python=False) and activating poetry's one with poetry shell.
Question: I am trying to use Mutagen for changing ID3 (version 2.3) cover art for a bunch of MP3 files in the following way: from mutagen.mp3 import MP3 from mutagen.id3 import APIC file = MP3(filename) with open('Label.jpg', 'rb') as albumart: file.tags['APIC'] = APIC( encoding=3, mime='image/jpeg', type=3, desc=u'Cover', data=albumart.read() ) file.save(v2_version=3)  However, the file (or at least the APIC tag) remains unchanged, as checked by reading the tag back. In the system file explorer, the file does show an updated Date modified, however. How can I get Mutagen to update the cover art correctly?

Answer: I needed to set the cover to the "APIC:" tag, instead of the "APIC" tag (which I guess is how IDv2.3 is specified).
Question: Suppose I have a list [2, 4, 1, 3, 5]. I want to sort the list just from index 1 to the end, which gives me [2, 1, 3, 4, 5] How can I do it in Python? (No extra spaces would be appreciated)

Answer: TL;DR: Use sorted with a slicing assignment to keep the original list object without creating a new one: l = [2, 4, 1, 3, 5] l[1:] = sorted(l[1:]) print(l)  Output: [2, 1, 3, 4, 5]  Longer Answer: After the list is created, we will make a slicing assignment: l[1:] =  Now you might be wondering what does [1:], it is slicing the list and starts from the second index, so the first index will be dropped. Python's indexing starts from zero, : means get everything after the index before, but if it was [1:3] it will only get values that are in between the indexes 1 and 3, let's say your list is: l = [1, 2, 3, 4, 5]  If you use: print(l[1:])  It will result in: [2, 3, 4, 5]  And if you use: print(l[1:3])  It will result in: [2, 3]  About slicing, read more here if you want to. And after slicing we have an equal sign =, that just simply changes what's before the = sign to what's after the = sign, so in this case, we use l[1:], and that gives [2, 3, 4, 5], it will change that to whatever is after the = sign. If you use: l[1:] = [100, 200, 300, 400] print(l)  It will result in: [1, 100, 200, 300, 400]  To learn more about it check out this. After that, we got sorted, which is default builtin function, it simple sorts the list from small to big, let's say we have the below list: l = [3, 2, 1, 4]  If you use: print(sorted(l))  It will result in: [1, 2, 3, 4]  To learn more about it check this. After that we come back to our first topic about slicing, with l[1:], but from here you know that it isn't only used for assignments, you can apply functions to it and deal with it, like here we use sorted.
Question: I'm trying to figure out what is the reason for having None as the default value for dict.get but no default value (without specifying the default value) for dict.pop {}.get('my_key') # output: None {}.pop('my_key') # output: KeyError: 'my_key'  I was thinking that the reason for not having implicit default value for dict.pop is because you may have keys with value None so, in order to not get confused if your key is in the dictionary or not, an implicit default value for dict.pop doesn't make so much sense. But then again this explanation should be valid also for dict.get and isn't: {'my_key': None}.get('my_key') # output: None # but doesn't tell you if the key is truly in the dictionary or not 

Answer: My take on this is to allow a dict to be used with different contexts. With dict.get(key) we always get a value even if the key is not present in dict. The default value can be provided. No exception is raised. The dict is not changed. With dict.pop(key) we get a value only when the key is present in dict, otherwise an exception is raised. We may avoid the exception by providing a default value. The dict is changed. To test for the presence of key we use key in dict. For dict.pop, this gives a similar interface to what list.pop, set.pop and deque.pop provides. In short a good example of the "principle of least surprise" and the Zen of Python (import this) :)
Question: Post-install script with Python setuptools Exactly this question, but with Poetry and no Setuptools.  I want to run print('Installation finished, doing other things...')  when my package is installed. With Setuptools you could just modify setup.py, but in Poetry there specifically is no setup.py. What I actually want to do is generate a default .mypackage_config file and place it somewhere useful. I don't see how to do this without arbitrary code, but Poetry does not allow arbitrary code for installation. Is there any way to do this?

Answer: It's not currently possible (and probably won't ever be) The entire idea of poetry is that a package can be installed without running any arbitrary Python code. Because of that, custom post-install scripts will probably never exist (from the author of poetry, the link you gave in your question). What you could do instead You could use setuptools, and modify the setup.py script, but it's probably easier to remove the need for a post-install script by removing the need for a default config file. Most Python tools, such as black, tend to assume default config parameters, unless there are settings in a config file (such as a pyproject.toml file) that overrides them, e.g.: [tool.black] # specifies this next section is the config for black line-length = 88 # changes some configuration parameters from the defaults target-version = ['py36', 'py37']  Jupyter has a command jupyterhub --generate-config to generate a default config file (source). If your package is a library, not a command-line tool, i.e. you use it by import-ing it in your Python code, I'd recommend just passing in the configuration as arguments to the constructor/functions, since that's the standard way of passing config to a library. As an example, look at PySerial's API. You can configure the library by passing args to the constructor serial.Serial, e.g.: import serial with serial.Serial( '/dev/ttyUSB0', # first config param, the port 19200, # second config param, the baudrate timeout=5, # sets the timeout config param ) as ser: