eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a some nested loops (three total) where I'm trying to use numpy.einsum to speed up the calculations, but I'm struggling to get the notation correct. I managed to get rid of one loop, but I can't figure out the other two. Here's what I've got so far: import numpy as np import time def myfunc(r, q, f): nr = r.shape[0] nq = q.shape[0] y = np.zeros(nq) for ri in range(nr): for qi in range(nq): y[qi] += np.einsum('i,i',f[ri,qi]*f[:,qi],np.sinc(q[qi]*r[ri,:]/np.pi)) return y r = np.random.random(size=(1000,1000)) q = np.linspace(0,1,1001) f = np.random.random(size=(r.shape[0],q.shape[0])) start = time.time() y = myfunc(r, q, f) end = time.time() print(end-start)  While this was much faster than the original, this is still too slow, and takes about 30 seconds. Note the original without the einsum call was the following (which looks like it will take ~2.5 hours, didn't wait to find out for sure): def myfunc(r, q, f): nr = r.shape[0] nq = q.shape[0] y = np.zeros(nq) for ri in range(nr): for rj in range(nr): for qi in range(nq): y[qi] += f[ri,qi]*f[rj,qi]*np.sinc(q[qi]*r[ri,rj]/np.pi)) return y  Does anyone know how to get rid of these loops with an einsum, or any other tool for that matter?

Answer: Your function seems to be equivalent to the following: # this is so called broadcasting s = np.sinc(q * r[...,None]/np.pi) np.einsum('iq,jq,ijq->q',f,f,s)  Which took about 20 seconds on my system, with most of the time to allocate s. Let's test it for a small sample: np.random.seed(1) r = np.random.random(size=(10,10)) q = np.linspace(0,1,1001) f = np.random.random(size=(r.shape[0],q.shape[0])) (np.abs(np.einsum('iq,jq,ijq->q',f,f,s) - myfunc(r,q,f)) < 1e-6).all() # True  Since np.sinc is not a linear operator, I'm not quite sure how we can further reduce the run time.
Question: I need to find if items from a list appear in a string, and then add the items to a different list. This code works: data =[] line = 'akhgvfalfhda.dhgfa.lidhfalihflaih**Thing1**aoufgyafkugafkjhafkjhflahfklh**Thing2**dlfkhalfhafli...' _legal = ['thing1', 'thing2', 'thing3', 'thing4',...] for i in _legal: if i in line: data.append(i)  However, the code iterates over line (which could be long) multiple times- as many times as there are item in _legal (which could be a lot). That's too slow for me, and I'm searching for a way to do it faster. line doesn't have any specific format, so using .split() couldn't work, as far as I know. Edit: changed line so that it better represents the problems.

Answer: One way I could think of to improve is:  Get all unique lengths of the words in _legal Build a dictionary of words from line of those particular lengths using a sliding window technique. The complexity should be O( len(line)*num_of_unique_lengths ), this should be better than brute force. Now look for each thing in the dictionary in O(1).  Code: line = 'thing1 thing2 456 xxualt542l lthin. dfjladjfj lauthina ' _legal = ['thing1', 'thing2', 'thing3', 'thing4', 't5', '5', 'fj la'] ul = {len(i) for i in _legal} s=set() for l in ul: s = s.union({line[i:i+l] for i in range(len(line)-l)}) print(s.intersection(set(_legal)))  Output: {'thing1', 'fj la', 'thing2', 't5', '5'}
Question: I have a project with this setup.py file: import setuptools with open("README.md", "r") as fh: long_description = fh.read() setuptools.setup( name="", version="0.0.1", author="", author_email="", description="", long_description=long_description, long_description_content_type="text/markdown", packages=setuptools.find_packages(where="./src", exclude=("./tests",)), classifiers=[ "Programming Language :: Python :: 3", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", ], python_requires='>=3.8', )  This is my project directory structure (first two levels): $ tree -L 2 . ├── README.md ├── setup.py ├── src │ └── my_pkg └── tests ├── conftest.py ├── data ├── __init__.py ├── integration ├── __pycache__ └── unit  When I run any setuptools command, I get the following error: $ python setup.py build running build running build_py error: package directory 'my_pkg' does not exist  The same happens for other commands like python setup.py develop and python setup.py bdist-wheel. I suspect that it has to do with the src directory, as specified in the find_packages(where="./src") call in the setup.py. However, I've been following the documentation, and it does look the the my_pkg module is discovered at some point. Why does build_py fail to find it?

Answer: find_packages() automatically generates package names. That is, in your case all it does is generate ['my_pkg']. It doesn't actually tell setup() where to find my_pkg, just that it should expect to find a package called my_pkg somewhere. You have to separately tell setup() where it should look for packages. Is this confusing and counter intuitive? Yes. Anyway, you can tell setup() where to find my_pkg by using the package_dir argument. eg. package_dir={"":"src"}
Question: Problem I'm trying to load a file using PyTorch, but the error states archive/data.pkl does not exist. Code import torch cachefile = 'cacheddata.pth' torch.load(cachefile)  Output --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-4-8edf1f27a4bd> in <module> 1 import torch 2 cachefile = 'cacheddata.pth' ----> 3 torch.load(cachefile) ~/opt/anaconda3/envs/matching/lib/python3.8/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args) 582 opened_file.seek(orig_position) 583 return torch.jit.load(opened_file) --> 584 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args) 585 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) 586 ~/opt/anaconda3/envs/matching/lib/python3.8/site-packages/torch/serialization.py in _load(zip_file, map_location, pickle_module, **pickle_load_args) 837 838 # Load the data (which may in turn use `persistent_load` to load tensors) --> 839 data_file = io.BytesIO(zip_file.get_record('data.pkl')) 840 unpickler = pickle_module.Unpickler(data_file, **pickle_load_args) 841 unpickler.persistent_load = persistent_load RuntimeError: [enforce fail at inline_container.cc:209] . file not found: archive/data.pkl  Hypothesis I'm guessing this has something to do with pickle, from the docs:  This save/load process uses the most intuitive syntax and involves the least amount of code. Saving a model in this way will save the entire module using Python’s pickle module. The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. The reason for this is because pickle does not save the model class itself. Rather, it saves a path to the file containing the class, which is used during load time. Because of this, your code can break in various ways when used in other projects or after refactors.  Versions  PyTorch version: 1.6.0 Python version: 3.8.0 

Answer: Turned out the file was somehow corrupted. After generating it again it loaded without issue.
Question: I'm learning Apache Airflow to implement it at my workplace, I stumbled on a problem, when trying to pass parameter to function like this (I followed the documentation) from airflow import DAG import pendulum from datetime import datetime, timedelta from airflow.operators.python_operator import PythonOperator args = { "owner": "airflow", "start_date": pendulum.datetime(year=2020, month=10, day=5, tzinfo='Asia/Shanghai'), "retries": 5, "retry_delay": timedelta(minutes=3) } dag = DAG( "example_dag_v2", schedule_interval="@daily", default_args=args ) def my_mult_function(number): return number*number mult_task = PythonOperator( task_id = 'mult_task', provide_context=True, python_callable=my_mult_function, op_kwargs={'number': 5}, dag = dag ) mult_task  I keep getting this error  TypeError: my_mult_function() got an unexpected keyword argument 'conf'  where did I do wrong ? Solution: so i found the solution but still dont understand why the solution is def my_mult_function(number, **kwargs): return number*number  i passed **kwargs on the parameters, and it works! But i still dont understand why i need to pass the **kwargs ?

Answer: You have set provide_context=True so PythonOperator will send the execute context to your python_callable. So a generic catch all keyword arguments, **kwargs fixes the issue. https://github.com/apache/airflow/blob/v1-10-stable/airflow/operators/python_operator.py#L108. If you are not going to use anything from the context then set provide_context=False.
Question: I'm trying to download a large Dropbox folder with a bunch of subfolders to an external harddrive using the scripts at: http://tilburgsciencehub.com/examples/dropbox/ import dropbox from get_dropbox import get_folders, get_files from dropbox.exceptions import ApiError, AuthError # read access token access_token = "sl.superlongstring" print('Authenticating with Dropbox...') try: dbx = dropbox.Dropbox(access_token, scope=['files.content.read', 'files.metadata.read']) print('...authenticated with Dropbox owned by ' + dbx.users_get_current_account().name.display_name) except AuthError as err: print(err)  no errors here , correctly displays my name. try: folders=get_folders(dbx, '/Audioboeken') print(folders) download_dir = r'L:\\00 Audioboeken' print('Obtaining list of files in target directory...') get_files(dbx, folder_id, download_dir) except ApiError as err: print(err) except AuthError as autherr: print(autherr)  This errors:  dropbox.exceptions.AuthError: AuthError('randomstringofnumbers, AuthError('missing_scope', TokenScopeError(required_scope='files.metadata.read')))  I've tried adding scope to the login request, but that doesn't seem to help..(not sure if I did that correctly) App Permissions: The checkbox for files.content.read and files.metadata.read are checked.

Answer: The 'missing_scope' error indicates that while the app is permitted to use that scope, the particular access token you're using to make the API call does not have that scope granted. Adding a scope to your app via the App Console does not retroactively grant that scope to existing access tokens. That being the case, to make any API calls that require that scope, you'll need to get a new access token with that scope.
Question: I need a way to reverse my key values pairing. Let me illustrate my requirements.  dict = {1: (a, b), 2: (c, d), 3: (e, f)}  I want the above to be converted to the following: dict = {1: (e, f), 2: (c, d), 3: (a, b)} 

Answer: You just need: new_dict = dict(zip(old_dict, reversed(old_dict.values())))  Note, prior to Python 3.8, where dict_values objects are not reversible, you will need something like: new_dict = dict(zip(old_dict, reversed(list(old_dict.values()))))
Question: Here is what I did: ❯ pip freeze aiohttp @ file:///Users/aiven/Library/Caches/pypoetry/artifacts/50/32/0b/b64b02b6cefa4c089d84ab9edf6f0d960ca26cfbe57fe0e693a00912da/aiohttp-3.6.2-py3-none-any.whl async-timeout @ file:///Users/aiven/Library/Caches/pypoetry/artifacts/0d/5d/3e/630122e534c1b25e36c3142597c4b0b2e9d3f2e0a9cea9f10ac219f9a7/async_timeout-3.0.1-py3-none-any.whl attrs @ file:///Users/aiven/Library/Caches/pypoetry/artifacts/7f/e7/44/32ca3c400bb4d8a2f1a91d1d3f22bbaee2f4727a037aad898fbf5d36ce/attrs-20.2.0-py2.py3-none-any.whl chardet @ file:///Users/aiven/Library/Caches/pypoetry/artifacts/c2/02/35/0d93b80c730b360c5e3d9bdc1b8d1929dbd784ffa8e3db025c14c045e4/chardet-3.0.4-py2.py3-none-any.whl ...  Version of pip: ❯ pip -V pip 20.2.3 from /Users/aiven/projects/foobar/venv/lib/python3.8/site-packages/pip (python 3.8)  I expected something like this: > pip freeze foo==1.1.1 bar==0.2.1  pip freeze -h wasn't very helpful... For context: I installed packages into virtualenv using poetry.

Answer: This seems to come from the changes to support PEP 610. In particular refer to the Freezing an environment section. The notion of what "freezing" entails has been expanded to include preserving direct url sources for packages that were installed with direct origins. Poetry, with 1.1.0 has introduced a new installer that now handles discovery and download of artifacts for dependencies. This is different to the behaviour in 1.0.10 which simply let pip handle discover and download of required artifacts (wheels). This means that, now packages are installed using direct URL origins. This causes pip freeze to use direct reference format as specified in PEP 508 (eg: package @ file://../package.whl). For those interested, the url in question will be saved in <package>-<version>.dist-info/direct_url.json in the virtual env's site directory. You can get the old format output (not sure if this will change in the future), using the following command. pip --disable-pip-version-check list --format=freeze
Question: so far my dataframe looks like this: ID Area Stage 1 P X 2 Q X 3 P X 4 Q Y  I would like to replace the area 'Q' with 'P' for every row where the Stage is equal to 'X'. So the result should look like: ID Area Stage 1 P X 2 P X 3 P X 4 Q Y  I tried: data.query('Stage in ["X"]')['Area']=data.query('Stage in ["X"]')['Area'].replace('Q','P')  It does not work. Help is appreciated! :)

Answer: You can use loc to specify where you want to replace, and pass the replaced series to the assignment: df.loc[df['Stage']=='X', 'Area'] = df['Area'].replace('Q','P')  Output:  ID Area Stage 0 1 P X 1 2 P X 2 3 P X 3 4 Q Y
Question: I'm using the python-telegram-bot library to write a bot in Python that sends URLs into a channel where the bot is administrator. Now, I would like to have the bot reading, let's say, the last 5 messages (I don't really care about the number as I just need to read the message on the chat) and store them into a list in the code for further elaborations. I already have my bot working with: bot = telegram.Bot(token='mytoken') bot.sendMessage(chat_id='@mychatid', text=entry.link)  But I can't find a bot.getLastMessage or bot.getMessage kind of class into the python-telegram-bot library. In case there's already no written class that does that, how can I implement it via the Telegram API as I'm a bit of a beginner when it comes to API implementation? Thanks.

Answer: That's not possible in Bots unfortunately. Here you can find all available methods (that python-telegram-bot invokes behind the scenes) and there's no such method available to fetch messages on demand. The closest you can get through the api is getChat (which would return the pinned_message in that chat). What you can do in this case is, store the messages the bot sends as well as the message updates the bot receives (by setting up a handler) in some storage (database) and fetch from there later on.
Question: This seems like it should be simple enough, but haven't been able to find a working example of how to approach this. Simply put I am generating a JSON file based on a list that a script generates. What I would like to do, is use some variables to run the dump() function, and produce a json file into specific folders. By default it of course dumps into the same place the .py file is located, but can't seem to find a way to run the .py file separately, and then produce the JSON file in a new folder of my choice: import json name = 'Best' season = '2019-2020' blah = ['steve','martin'] with open(season + '.json', 'w') as json_file: json.dump(blah, json_file)  Take for example the above. What I'd want to do is the following:  Take the variable 'name', and use that to generate a folder of the same name inside the folder the .py file is itself. This would then place the JSON file, in the folder, that I can then manipulate.   Right now my issue is that I can't find a way to produce the file in a specific folder. Any suggestions, as this does seem simple enough, but nothing I've found had a method to do this. Thanks!

Answer: Python's pathlib is quite convenient to use for this task: import json from pathlib import Path data = ['steve','martin'] season = '2019-2020'  Paths of the new directory and json file: base = Path('Best') jsonpath = base / (season + ".json")  Create the directory if it does not exist and write json file: base.mkdir(exist_ok=True) jsonpath.write_text(json.dumps(data))  This will create the directory relative to the directory you started the script in. If you wanted a absolute path, you could use Path('/somewhere/Best'). If you wanted to start the script while beeing in some other directory and still create the new directory into the script's directory, use: Path(__file__).resolve().parent / 'Best'.
Question: I have a Pandas dataframe (countries) and need to get specific index value. (Say index 2 => I need Japan)  I used iloc, but i got the data (7.542) return countries.iloc[2] 7.542 

Answer: call the index directly return countries.index[2]  but what you post here looks like a pandas dataframe instead of a series - if that's the case do countries['Country_Name'].iloc[2]
Question: When I run the command flower -A main --port=5555  Flower doesn't work, the error is: > ImportError: cannot import name 'Command' from 'celery.bin.base'  Any ideas? Main is a Django Project

Answer: Flower is always lagging behind Celery, so if you use the latest Celery (they refactored the CLI) it will probably fail. Stick to 4.4.x until Flower catches up.
Question: I have a bunch of numbers in scientific notation and I would like to find their exponent. For example: >>> find_exp(3.7e-13) 13 >>> find_exp(-7.2e-11) 11  Hence I only need their exponent ignoring everything else including the sign. I have looked for such a way in python but similar questions are only for formatting purposes.

Answer: Common logarithm is what you need here, you can use log10 + floor: from math import log10, floor def find_exp(number) -> int: base10 = log10(abs(number)) return abs(floor(base10)) find_exp(3.7e-13) # 13 find_exp(-7.2e-11) # 11
Question: If I have local/global variable var of any type how do I get its name, i.e. string "var"? I.e. for some imaginary function or operator nameof() next code should work: var = 123 assert nameof(var) == "var"  There's .__name__ property for getting name of a function or a type object that variable holds value of, but is there anything like this for getting name of a variable itself? Can this be achieved without wrapping a variable into some magic object, as some libraries do in order to get variable's name? If not possible to achieve without magic wrappers then what is the most common/popular wrapping library used for this case?

Answer: You can do this with the package python-varname: https://github.com/pwwang/python-varname First run pip install varname. Then see the code below: from varname import nameof var = 123 name = nameof(var) #name will be 'var'
Question: Say I have an array like this: x = [1, 2, 3] [4, 5, 6] [7, 8, 9]  And I want to delete both the ith row and column. So if i=1, I'd create (with 0-indexing): [1, 3] [7, 9]  Is there an easy way of doing this with a one-liner? I know I can call np.delete() twice, but that seems a little unclean. It'd be exactly equivalent to np.delete(np.delete(x, idx, 0), idx, 1), where idx is the index of the row/column pair to delete - it'd just look cleaner.

Answer: In [196]: x = np.arange(1,10).reshape(3,3)  If you look at np.delete code, you'll see that it's python (not compiled) and takes different approaches depending on how the delete values are specified. One is to make a res array of right size, and copy two slices to it. Another is to make a boolean mask. For example: In [197]: mask = np.ones(x.shape[0], bool) In [198]: mask[1] = 0 In [199]: mask Out[199]: array([ True, False, True])  Since you are deleting the same row and column, use this indexing: In [200]: x[mask,:][:,mask] Out[200]: array([[1, 3], [7, 9]])  A 1d boolean mask like this can't be 'broadcasted' in the same ways a integer array can. We can do the 2d advanced indexing with: In [201]: idx = np.nonzero(mask)[0] In [202]: idx Out[202]: array([0, 2]) In [203]: np.ix_(idx,idx) Out[203]: (array([[0], [2]]), array([[0, 2]])) In [204]: x[np.ix_(idx,idx)] Out[204]: array([[1, 3], [7, 9]])  Actually ix_ can work directly from the boolean array(s): In [207]: np.ix_(mask,mask) Out[207]: (array([[0], [2]]), array([[0, 2]]))  This isn't a one-liner, but it probably is faster than the double delete, since it strips off all the extra baggage that the more general function requires.
Question: Suppose, I have a code snippet as foo = SomeClass() bar = foo[1:999].execute() To test this, I have tried something as foo_mock = Mock() foo_mock[1:999].execute() Unfortunately, this raised an exception,  TypeError: 'Mock' object is not subscriptable  So, How can I create a subscriptable Mock object?

Answer: Just use a MagicMock instead. >>> from unittest.mock import Mock, MagicMock >>> Mock()[1:999] TypeError: 'Mock' object is not subscriptable >>> MagicMock()[1:999] <MagicMock name='mock.__getitem__()' id='140737078563504'>  It's so called "magic" because it supports __magic__ methods such as __getitem__.
Question: Where is python intrepreter located in virtual environment ? I am making a GUI project and I stuck while finding the python interpreter in my virtual environment. 

Answer: Execute next code and it will print location of your python interpreter. import sys print(sys.executable)
Question: I have a simple web app built with Django & running with Gunicorn with Nginx. When I open HTOP, I see there are so many processes & threads spawn -- for a single tutorial app that just displays a login form. See screenshot of HTOP below:  Why are there so many of them open for such a simple app? Here is my configuration """gunicorn WSGI server configuration.""" from multiprocessing import cpu_count from os import environ def max_workers(): return cpu_count() * 2 + 1 max_requests = 1000 worker_class = 'gevent' workers = max_workers()  Thanks

Answer: That's because of gunicorn's design:  Gunicorn is based on the pre-fork worker model. This means that there is a central master process that manages a set of worker processes. The master never knows anything about individual clients. All requests and responses are handled completely by worker processes.  That means that gunicorn will spawn as many processes and threads as it needs, depending on the configuration and type of the workers set.