eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm trying to run a PyTorch model in a Django app. As it is not recommended to execute the models (or any long-running task) in the views, I decided to run it in a Celery task. My model is quite big and it takes about 12 seconds to load and about 3 seconds to infer. That's why I decided that I couldn't afford to load it at every request. So I tried to load it at settings and save it there for the app to use it. So my final scheme is:  When the Django app starts, in the settings the PyTorch model is loaded and it's accessible from the app. When views.py receives a request, it delays a celery task The celery task uses the settings.model to infer the result  The problem here is that the celery task throws the following error when trying to use the model [2020-08-29 09:03:04,015: ERROR/ForkPoolWorker-1] Task app.tasks.task[458934d4-ea03-4bc9-8dcd-77e4c3a9caec] raised unexpected: RuntimeError("Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method") Traceback (most recent call last): File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task R = retval = fun(*args, **kwargs) File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__ return self.run(*args, **kwargs) /*...*/ File "/home/ubuntu/anaconda3/envs/tensor/lib/python3.7/site-packages/torch/cuda/__init__.py", line 191, in _lazy_init "Cannot re-initialize CUDA in forked subprocess. " + msg) RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method  Here's the code in my settings.py loading the model: if sys.argv and sys.argv[0].endswith('celery') and 'worker' in sys.argv: #In order to load only for the celery worker import torch torch.cuda.init() torch.backends.cudnn.benchmark = True load_model_file()  And the task code @task def getResult(name): print("Executing on GPU:", torch.cuda.is_available()) if os.path.isfile(name): try: outpath = model_inference(name) os.remove(name) return outpath except OSError as e: print("Error", name, "doesn't exist") return ""  The print in the task shows "Executing on GPU: true" I've tried setting torch.multiprocessing.set_start_method('spawn') in the settings.py before and after the torch.cuda.init() but it gives the same error.

Answer: Setting this method works as long as you're also using Process from the same library. from torch.multiprocessing import Pool, Process  Celery uses "regular" multiprocessing library, thus this error. If I were you I'd try either:  run it single threaded to see if that helps run it with eventlet to see if that helps read this
Question: I just got started with using the DropBox API for Python - i want to use it to store files that my Discord Bot previously downloaded, but even following the official tutorial 1:1 i cant get it to just read and write files. I registered the app and generated an access token, and it always tells me dropbox.exceptions.AuthError: AuthError('09d729accff6a6d8fa601154df010b0b', AuthError('missing_scope', TokenScopeError(required_scope='files.metadata.read'))) when i try to read dbx.files_list_folder('').entries for example. I checked the permissions and saw that apparently, the access token has no permissions yet, so i checked the right permissions. However, i cannot generate a new access token with the new permissions now, because it says `You must be a team administrator to perform this operation.´ I think i am missunderstanding something here, but i dont know why this basic example that i got from the offical site wont work.

Answer: Regarding the 'missing_scope' error: You're correct, the app and access token need the particular scope required by the route in order to access the route. Note that just enabling a particular scope for an app, via the App Console, does not retroactively add authorization for that scope to existing access tokens though. So, you'd need to process the authorization flow (either the OAuth flow or using the "Generate" button) again to authorize an access token with any newly enabled scopes, as you attempted. Regarding the "You must be a team administrator to perform this operation" error: That indicates that you selected one or more "team" scopes, which can only be authorized by a team admin, but you're not an admin on a Business team. In that case, you'd need to remove any team scopes to connect the app. Also, note that the 'files.permanent_delete' scope automatically requires the 'team_data.member' scope, so you can't use 'files.permanent_delete' from a non-team account, and so enabling the the 'files.permanent_delete' scope will also cause that. (Edit: Fixed: There's also a current issue that can prevent you from deselecting the 'team_data.member' scope if your app uses the "app folder" access type. We're working on fixing that, but you may need to re-create the app if you're in that state and want to use it before the fix is out.)
Question: I am using SetupTools to build a package of my own. In INSTALL_REQUIRES in setup.py I have the following dependencies: ... INSTALL_REQUIRES = [ 'ray>=0.8.7', 'pyyaml>=5.3', ] setup(name=PACKAGE_NAME, version=VERSION, description=DESCRIPTION, long_description=LONG_DESCRIPTION, long_description_content_type=LONG_DESC_TYPE, author=AUTHOR, license=LICENSE, author_email=AUTHOR_EMAIL, url=URL, install_requires=INSTALL_REQUIRES, packages=find_packages() )  When I run pip3 install -i https://test.pypi.org/simple/ r3po==0.0.6, I get the following error: ERROR: Could not find a version that satisfies the requirement pyyaml>=5.3 (from r3po==0.0.6) (from versions: 3.11) ERROR: No matching distribution found for pyyaml>=5.3 (from r3po==0.0.6)  However, pip3 search pyyaml shows me that PyYAML is definitely there: PyYAML (5.3.1) - YAML parser and emitter for Python and pip3 install pyyaml (in a .venv that has pyyaml installed) gives Requirement already satisfied: pyyaml in /home/lieu/dev/inzura-clustering-project/.venv/lib/python3.8/site-packages (5.3.1) And before you ask --- yes, I've tried all combinations of PyYAML, pyyaml, 5.3.1, 5.3, and so on, but nothing has worked. I even tried to remove the version requirement (so INSTALL_REQUIRES=['ray>=0.87','pyyaml']) but this leads to another error: ERROR: Command errored out with exit status 1: command: /home/lieu/dev/r3po/sample/.venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-w32z04mo/pyyaml/setup.py'"'"'; __file__='"'"'/tmp/pip-install-w32z04mo/pyyaml/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-w32z04mo/pyyaml/pip-egg-info cwd: /tmp/pip-install-w32z04mo/pyyaml/ Complete output (7 lines): running egg_info creating /tmp/pip-install-w32z04mo/pyyaml/pip-egg-info/PyYAML.egg-info writing /tmp/pip-install-w32z04mo/pyyaml/pip-egg-info/PyYAML.egg-info/PKG-INFO writing dependency_links to /tmp/pip-install-w32z04mo/pyyaml/pip-egg-info/PyYAML.egg-info/dependency_links.txt writing top-level names to /tmp/pip-install-w32z04mo/pyyaml/pip-egg-info/PyYAML.egg-info/top_level.txt writing manifest file '/tmp/pip-install-w32z04mo/pyyaml/pip-egg-info/PyYAML.egg-info/SOURCES.txt' error: package directory 'lib3/yaml' does not exist 

Answer: Anthony Sottile's suggestion to use --extra-index-url worked for me.
Question: Take python 3.6.x for example. The last windows installer for python 3.6.x is 3.6.8: no more installers for 3.6x version that comes later (see https://www.python.org/downloads/windows/) 3.6.8 happens to be the last maintenance release of python3.6, I don't know if it is somehow related to not propose a package installer for windows but only sources. Practical problem here: How should I proceed to install 3.6.12 on Windows? Please don’t simply advice « Install 3.7 or 3.8, it is more recent ». I know that 3.6 is not the latest, but sometimes you have to stick with a particular version for support or compatibility. Since I have to use 3.6.x, I am looking for the latest version available in this branch (currently 3.6.12) to still benefit from security patches. This gives two path:  install 3.6.8 with MSI installer then upgrade to 3.6.12 from source, install 3.6.12 from source.  What are the steps involved for option 1 or 2?

Answer: It is possible to create your own MSI installer from the source distributions at https://www.python.org/downloads/source/. This is what I did to install Python 3.6.12 on my Windows machine. In each source distribution, the files at PCBuild/readme.txt and Tools/msi/README.txt provide guidance for how to build your own Python installer. If you have not built Python from source before on Windows, this may be a challenge to set up. If you do not want to build the installer yourself, you can download unofficial installers from https://github.com/adang1345/PythonWindows. These are the installers that I built recently as part of a personal project.
Question: I want to make a script that shows the channels that i joined and then leave all of it with this example: from telethon.tl.functions.channels import LeaveChannelRequest await client(LeaveChannelRequest(input_channel)) 

Answer: In order to leave all the channels you're in, you have to fetch all the channels from the dialogs list and then just delete them. Here is a snippet. async for dialog in client.iter_dialogs(): if not dialog.is_group and dialog.is_channel: await dialog.delete()
Question: I found out that I could use Python to create a serverless function inside a Next.js project. Once deployed to Vercel, it will get converted into a serverless function. I went through the docs and found a simple example that outputs the date: from http.server import BaseHTTPRequestHandler from datetime import datetime class handler(BaseHTTPRequestHandler): def do_GET(self): self.send_response(200) self.send_header('Content-type', 'text/plain') self.end_headers() self.wfile.write(str(datetime.now().strftime('%Y-%m-%d %H:%M:%S')).encode()) return  They offer a live working example here. Apparently all that is needed is to place the file date.py inside the api folder of a bootstrapped Next.js project and you're off to the races. When deployed, Vercel will detect the Python file and serve it as a serverless function. The deploy succeeded and I placed the file inside the pages/api folder as required. However, the function is never picked up (image below):  Older versions apparently required the configuration of serverless functions by adding a vercel.json file. But this doesn't seem necessary now. What am I missing?

Answer: After going over the FAQs. I found an entry named Unmatched Function Pattern, it states:  the functions property uses a glob pattern for each key. This pattern must match Serverless Function source files within the api directory.  It also mentions:  if you'd like to use a Serverless Function that isn't written with Node.js in combination with Next.js, you can place it in the api directory (provided by the platform), since pages/api (provided by Next.js) only supports JavaScript.  I think that this needs to be clarified a bit. There is indeed an default api folder when you bootstrap a Next.js project with create-next-app, but it's created inside the pages directory. If you follow the example they give, you might just go ahead and create a serverless function in a supported language (other than JavaScript) inside the pages/api directory and wonder why Vercel doesn't pick it up when you deploy. In short, if you're using another language to write a serverless function inside a Next.js project. Be sure to place it inside an api folder that sits in the root directory of the project (if there's none, create one). Thanks to @evgenifotia for the suggestion, it pointed me in the right direction and helped me solve this issue. Note: You can only have a single api directory that houses serverless functions. Either you have a pages/api directory or a api directory in the root folder, having both in a single project is not supported.
Question: I have a file in my project which I would like to compile for performance reasons:  mylibrary/myfile.py  How to achieve this with Poetry?

Answer: There is an undocumented feature in Poetry. Add this to your pyproject.toml: [tool.poetry] ... build = 'build.py' [build-system] requires = ["poetry>=0.12", "cython"] build-backend = "poetry.masonry.api"  What this does is runs the build.py:build() function inside the implicitly generated setup.py. This is where we build. So, create a build.py that provides the build() function: import os # See if Cython is installed try: from Cython.Build import cythonize # Do nothing if Cython is not available except ImportError: # Got to provide this function. Otherwise, poetry will fail def build(setup_kwargs): pass # Cython is installed. Compile else: from setuptools import Extension from setuptools.dist import Distribution from distutils.command.build_ext import build_ext # This function will be executed in setup.py: def build(setup_kwargs): # The file you want to compile extensions = [ "mylibrary/myfile.py" ] # gcc arguments hack: enable optimizations os.environ['CFLAGS'] = '-O3' # Build setup_kwargs.update({ 'ext_modules': cythonize( extensions, language_level=3, compiler_directives={'linetrace': True}, ), 'cmdclass': {'build_ext': build_ext} })  Now, when you do poetry build, nothing happens. But if you install this package elsewhere, it gets compiled. You can also build it manually with: $ cythonize -X language_level=3 -a -i mylibrary/myfile.py  Finally, it seems that you can't publish binary packages to PyPi. The solution is to limit your build to "sdist": $ poetry build -f sdist $ poetry publish
Question: I have a TF-IDF matrix of shape (149,1001). What is want is to compute the cosine similarity of last columns, with all columns Here is what I did from numpy import dot from numpy.linalg import norm for i in range(mat.shape[1]-1): cos_sim = dot(mat[:,i], mat[:,-1])/(norm(mat[:,i])*norm(mat[:,-1])) cos_sim  But this loop is making it slow. So, is there any efficient way? I want to do with numpy only

Answer: Leverage 2D vectorized matrix-multiplication Here's one with NumPy using matrix-multiplication on 2D data - p1 = mat[:,-1].dot(mat[:,:-1]) p2 = norm(mat[:,:-1],axis=0)*norm(mat[:,-1]) out1 = p1/p2  Explanation : p1 is the vectorized equivalent of looping of dot(mat[:,i], mat[:,-1]). p2 is of (norm(mat[:,i])*norm(mat[:,-1])). Sample run for verification - In [57]: np.random.seed(0) ...: mat = np.random.rand(149,1001) In [58]: out = np.empty(mat.shape[1]-1) ...: for i in range(mat.shape[1]-1): ...: out[i] = dot(mat[:,i], mat[:,-1])/(norm(mat[:,i])*norm(mat[:,-1])) In [59]: p1 = mat[:,-1].dot(mat[:,:-1]) ...: p2 = norm(mat[:,:-1],axis=0)*norm(mat[:,-1]) ...: out1 = p1/p2 In [60]: np.allclose(out, out1) Out[60]: True  Timings - In [61]: %%timeit ...: out = np.empty(mat.shape[1]-1) ...: for i in range(mat.shape[1]-1): ...: out[i] = dot(mat[:,i], mat[:,-1])/(norm(mat[:,i])*norm(mat[:,-1])) 18.5 ms ± 977 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [62]: %%timeit ...: p1 = mat[:,-1].dot(mat[:,:-1]) ...: p2 = norm(mat[:,:-1],axis=0)*norm(mat[:,-1]) ...: out1 = p1/p2 939 µs ± 29.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) # @yatu's soln In [89]: a = mat In [90]: %timeit cosine_similarity(a[None,:,-1] , a.T[:-1]) 2.47 ms ± 461 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  Further optimize on norm with einsum Alternatively, we could compute p2 with np.einsum. So, norm(mat[:,:-1],axis=0) could be replaced by : np.sqrt(np.einsum('ij,ij->j',mat[:,:-1],mat[:,:-1]))  Hence, giving us a modified p2 : p2 = np.sqrt(np.einsum('ij,ij->j',mat[:,:-1],mat[:,:-1]))*norm(mat[:,-1])  Timings on same setup as earlier - In [82]: %%timeit ...: p1 = mat[:,-1].dot(mat[:,:-1]) ...: p2 = np.sqrt(np.einsum('ij,ij->j',mat[:,:-1],mat[:,:-1]))*norm(mat[:,-1]) ...: out1 = p1/p2 607 µs ± 132 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  30x+ speedup over loopy one!
Question: I have data as follows: ID Gender Country ... 1 Male UK 2 Female US 3 Male NZ 4 Female UK ...  There are only 2 options for gender and 3 for country. I would like to create a seperate pie chart for both "Gender" and "Country" to show how many times each option shows up in the data but I'm quite confused about how to do so. The data is stored in a pandas dataframe. Any and all help is much appreciated!

Answer: Here is an approach using pandas: import pandas as pd import numpy as np from matplotlib import pyplot as plt def label_function(val): return f'{val / 100 * len(df):.0f}\n{val:.0f}%' N = 50 df = pd.DataFrame({'country': np.random.choice(['UK', 'US', 'NZ'], N), 'gender': np.random.choice(['Male', 'Female'], N)}) fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(10, 5)) df.groupby('country').size().plot(kind='pie', autopct=label_function, textprops={'fontsize': 20}, colors=['tomato', 'gold', 'skyblue'], ax=ax1) df.groupby('gender').size().plot(kind='pie', autopct=label_function, textprops={'fontsize': 20}, colors=['violet', 'lime'], ax=ax2) ax1.set_ylabel('Per country', size=22) ax2.set_ylabel('Per gender', size=22)plt.tight_layout() plt.show()   PS: To just show the percentage, use autopct='%1.0f%%'.
Question: We have a machine learning classifier model that we have trained with a pandas dataframe and a standard sklearn pipeline (StandardScaler, RandomForestClassifier, GridSearchCV etc). We are working on Databricks and would like to scale up this pipeline to a large dataset using the parallel computation spark offers. What is the quickest way to convert our sklearn pipeline into something that computes in parallel? (We can easily switch between pandas and spark DFs as required.) For context, our options seem to be:  Rewrite the pipeline using MLLib (time-consuming) Use a sklearn-spark bridging library  On option 2, Spark-Sklearn seems to be deprecated, but Databricks instead recommends that we use joblibspark. However, this raises an exception on Databricks: from sklearn import svm, datasets from sklearn.model_selection import GridSearchCV from joblibspark import register_spark from sklearn.utils import parallel_backend register_spark() # register spark backend iris = datasets.load_iris() parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]} svr = svm.SVC(gamma='auto') clf = GridSearchCV(svr, parameters, cv=5) with parallel_backend('spark', n_jobs=3): clf.fit(iris.data, iris.target)  raises py4j.security.Py4JSecurityException: Method public int org.apache.spark.SparkContext.maxNumConcurrentTasks() is not whitelisted on class class org.apache.spark.SparkContext 

Answer: According to the Databricks instructions (here and here), the necessary requirements are:  Python 3.6+ pyspark>=2.4 scikit-learn>=0.21 joblib>=0.14  I cannot reproduce your issue in a community Databricks cluster running Python 3.7.5, Spark 3.0.0, scikit-learn 0.22.1, and joblib 0.14.1: import sys import sklearn import joblib spark.version # '3.0.0' sys.version # '3.7.5 (default, Nov 7 2019, 10:50:52) \n[GCC 8.3.0]' sklearn.__version__ # '0.22.1' joblib.__version__ # '0.14.1'  With the above settings, your code snippet runs smoothly, and produces indeed a classifier clf as: GridSearchCV(cv=5, error_score=nan, estimator=SVC(C=1.0, break_ties=False, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False), iid='deprecated', n_jobs=None, param_grid={'C': [1, 10], 'kernel': ('linear', 'rbf')}, pre_dispatch='2*n_jobs', refit=True, return_train_score=False, scoring=None, verbose=0)  as does the alternative example from here: from sklearn.utils import parallel_backend from sklearn.model_selection import cross_val_score from sklearn import datasets from sklearn import svm from joblibspark import register_spark register_spark() # register spark backend iris = datasets.load_iris() clf = svm.SVC(kernel='linear', C=1) with parallel_backend('spark', n_jobs=3): scores = cross_val_score(clf, iris.data, iris.target, cv=5) print(scores)  giving [0.96666667 1. 0.96666667 0.96666667 1. ]
Question: When using python's unittest subtest, I am confused regarding how parameters are named and scoped within the sub-test. The canonical example given in the link above seems to imply that the parameters used within the with self.subtest() clause can be passed as keyword arguments to subTest(). For reference, the example shown is this: class NumbersTest(unittest.TestCase): def test_even(self): """ Test that numbers between 0 and 5 are all even. """ for i in range(0, 6): with self.subTest(i=i): self.assertEqual(i % 2, 0)  It is using an ambiguous convention of naming the inner-scoped variable the same as the parameter (i=i). I took this to mean that keyword argument name is taken as the inner scoped variable name. However, when I tried to create my own test, I found that both PyCharm and the python interpreter complained of unresolved references if the keyword argument was not named precisely the same as the outer-scoped variable used as the parameter input. i.e.: class NumbersTest(unittest.TestCase): def test_even(self): """ Test that numbers between 0 and 5 are all even. """ for i in range(0, 6): with self.subTest(num=i): # <-- Renamed keyword argument parameter self.assertEqual(num % 2, 0) # < -- Results in unresolved reference error "num"  How does one pass parameters into the subtest? How are they named and referenced within the subtest code block?

Answer: It seems to me that it would be nice if this was expounded on a little bit more in the docs, but the API for subTest(msg=None, **params) states:  ...msg and params are optional, arbitrary values which are displayed whenever a subtest fails, allowing you to identify them clearly.  So it seems that the keyword arguments passed in **params are used only for test identification when printing test status to the console. They don't get passed in as parameters to the code block in any fashion.
Question: I need to make a program that alerts me with a windows notification, and I found out that this can be simply done with the following code. I don't care what library I use from win10toast import ToastNotifier toast = ToastNotifier() toast.show_toast("alert","text")  This code gives that following alert  However, I want there to be a button on the notification so I can click it and it will lead me to a url.  Like this example. Is this possible? I just found this website about toast contents can anyone help me use this with python?

Answer: This type of behavior is not supported in the currently released version of Windows-10-Toast-Notifications. However, a contributor created a pull request that adds functionality for a callback_on_click parameter that will call a function when the notification is clicked. This has yet to be merged into the master branch, and given how long it's been since the library has been updated, I wouldn't count on it happening anytime soon. However, you can still install this modified version of the library to make use of this feature:  First, you'll need to uninstall the current version of win10toast from your environment (e.g., pip uninstall win10toast). Next, you'll need to install the modified version (e.g., pip install git+https://github.com/Charnelx/Windows-10-Toast-Notifications.git#egg=win10toast).  Then, you can create a toast like this: toast.show_toast(title="Notification", msg="Hello, there!", callback_on_click=your_callback_function)  A complete working example: from win10toast import Toast toast = ToastNotifier() toast.show_toast(title="Notification", msg="Hello, there!", callback_on_click=lambda: print("Clicked!"))  When you click on the notification, you should see "Clicked!" appear in the Python console. Important: This will only work if you're using the modified version of the library I mentioned above. Otherwise you will get the error: TypeError: show_toast() got an unexpected keyword argument 'callback_on_click'.