eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to run a notebook in databricks from another notebook using %run. Also I want to be able to send the path of the notebook that I'm running to the main notebook as a parameter. The reason for not using dbutils.notebook.run is that I'm storing nested dictionaries in the notebook that's called and I wanna use them in the main notebook. I'm looking for Something like: path = "/References/parameterDefinition/schemaRepository"  %run <path variable> 

Answer: Unfortunately it's impossible to pass the path in %run as variable. You can pass variable as parameter only, and it's possible only in combination with with widgets - you can see the example in this answer. In this case you can have all your definitions in one notebook, and depending on the passed variable you can redefine the dictionary. There will be a new functionality coming in the next months (approximately, see public roadmap webinar for more details) that will allow to import notebooks as libraries using the import statement. Potentially you can emulate the same functionality by exporting the notebook into the file on disk using the Export command of Workspace API, decoding the data & importing file's content, for example, if you have notebook called module1 with content my_cool_dict = {"key1": "abc", "key2": 123}  then you can import it as following: import requests import base64 import os api_url = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get() host_token = "your_PAT_token" path = "/Users/..../module1" # fetch notebook response = requests.get(f"{api_url}/api/2.0/workspace/export", json = {"format": "SOURCE", "path": path}, headers={"Authorization": f"Bearer {host_token}"} ).json() # decode base64 encoded content data = base64.b64decode(response["content"].encode("ascii")) # write the file & __init__.py, so directory will considered a module dir = os.path.join("/tmp","my_modules") if not os.path.exists(dir): os.mkdir(dir) with open(os.path.join(dir, os.path.split(path)[-1]+".py"), "wb") as f: f.write(data) with open(os.path.join(dir, "__init__.py"), "wb") as f: f.write("\n".encode("ascii")) # add our directory into system path import sys sys.path.append(dir) # import notebook from module1 import my_cool_dict  and see that we got our variable:
Question: I'm looking to copy the docstring of a function in the same file by name (with a decorator). I can easily do it with a function that is out of the current module, but I'm a bit confused when it comes to the same module (or the same class more specifically) Here's what I have so far: import inspect def copy_doc(func_name: str): def wrapper(func): doc = ... # get doc from function that has the name as func_name func.__doc__ = doc return func retun wrapper  I'm looking for something that can do two of these examples: Ex 1: def this() -> None: """Fun doc string""" return @copy_doc('this') def that() -> None: return print(that.__doc__)  Ex 2: class This: def foo(self) -> None: """Fun doc string""" return None @copy_doc('foo') def bar(self) -> None: return None print(This().bar.__doc__)  Any fun ideas?

Answer: After some testing and experimentation, I learned you could directly reference the function in a given class. * Note ParamSpec and TypeVar are to keep the correct signature of the wrapped function, you can remove all the annotations if you do not need them. from typing import Callable, TypeVar, Any, TypeAlias from typing_extensions import ParamSpec T = TypeVar('T') P = ParamSpec('P') WrappedFuncDeco: TypeAlias = Callable[[Callable[P, T]], Callable[P, T]] def copy_doc(copy_func: Callable[..., Any]) -> WrappedFuncDeco[P, T]: """Copies the doc string of the given function to another. This function is intended to be used as a decorator. .. code-block:: python3 def foo(): '''This is a foo doc string''' ... @copy_doc(foo) def bar(): ... """ def wrapped(func: Callable[P, T]) -> Callable[P, T]: func.__doc__ = copy_func.__doc__ return func return wrapped class Cas: def foo(self) -> None: """This is the foo doc string.""" return @copy_doc(foo) def bar(self) -> None: return print(Cas.bar.__doc__) # >>> This is the foo doc string.
Question: I use aiohttp, pytest and pytest-cov to get coverage of my code. I want to get better coverage< but now I am a little bit stuck, because event simple code does not show 100% cov. For example this piece of code: @session_decorator() async def healthcheck(session, request): await session.scalar("select pg_is_in_recovery();") return web.json_response({"status": "ok"})  in coverage report it shows that line with return is not covered. I have read this link Code coverage for async methods. But this is for C# and I can not understand:  This can happen most commonly if the operation you're awaiting is completed before it's awaited.  My code to run tests: python3.9 -m pytest -vv --cov --cov-report xml --cov-report term-missing  My test for code above async def test_healthz_page(test_client): response = await test_client.get("/healthz") assert response.status == HTTPStatus.OK 

Answer: I had the same issue when testing FastAPI code using asyncio. The fix is to create or edit a .coveragerc at the root of your project with the following content: [run] concurrency = gevent  If you use a pyproject.toml you can also include this section in that file instead: [tool.coverage.run] concurrency = ["gevent"]  In addition, you have to pip install gevent. If using Poetry, run poetry add --group dev gevent. If the above doesn’t work, try using concurrency = thread,gevent instead (see this comment. Note it says to use --concurrency but this option is not available when you use pytest-cov; you must use .coveragerc).
Question: I'm a bit new in celery configs. I have a task named myapp.tasks.my_task for example. I can see myapp.tasks.my_task in registered tasks of celery when I use celery inspect registered. doesn't it mean that the task is successfully registered? why it raises the following error for it: KeyError celery.worker.consumer.consumer in on_task_received Received unregistered task of type 'my_app.tasks.my_task'. The message has been ignored and discarded. Did you remember to import the module containing this task? Or maybe you're using relative imports? Please see http://docs.celeryq.org/en/latest/internals/protocol.html for more information. The full contents of the message body was: '[[], {}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (77b)  there are also other tasks in my_app.tasks and they work correctly but only this task does not work and gets KeyError: @shared_task(queue='celery') def other_task(): """ WORKS """ ... @shared_task(queue='celery') def my_task(): """ DOES NOT WORK """ ... 

Answer: Back in the day, when I faced the problem a senior solved it for me in a mushroom management way unfortunately (see more about this anti-pattern here). I came back to this problem recently to figure out the solution in our own project domain. As Niel pointed in his/her solution, we were using celery_app.autodiscover_tasks() in our project and in that case we should import my_task in __init__.py of tasks package like bellow. from .some_tasks_file import my_task  Also, we used celery beat and the task defined inside app.conf.beat_schedule must have exact path to the function like bellow (even though the function is imported in __init__.py of the tasks package). app.conf.beat_schedule = { 'MY_TASK': { 'task': 'myapp.tasks.some_tasks_file.my_task', 'schedule': 60, # every minute }, }  Hope this would help people with the same celery configuration and problem.
Question: Im attempting to find model performance metrics (F1 score, accuracy, recall) following this guide https://machinelearningmastery.com/how-to-calculate-precision-recall-f1-and-more-for-deep-learning-models/ This exact code was working a few months ago but now returning all sorts of errors, very confusing since i havent changed one character of this code. Maybe a package update has changed things? I fit the sequential model with model.fit, then used model.evaluate to find test accuracy. Now i am attempting to use model.predict_classes to make class predictions (model is a multi-class classifier). Code shown below: model = Sequential() model.add(Dense(24, input_dim=13, activation='relu')) model.add(Dense(18, activation='relu')) model.add(Dense(6, activation='softmax')) model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) - history = model.fit(X_train, y_train, batch_size = 256, epochs = 10, verbose = 2, validation_split = 0.2) - score, acc = model.evaluate(X_test, y_test,verbose=2, batch_size= 256) print('test accuracy:', acc) - yhat_classes = model.predict_classes(X_test)  last line returns error "AttributeError: 'Sequential' object has no attribute 'predict_classes'" This exact code was working not long ago so struggling a bit, thanks for any help

Answer: This function was removed in TensorFlow version 2.6. According to the keras in rstudio reference update to predict_x=model.predict(X_test) classes_x=np.argmax(predict_x,axis=1)  Or use TensorFlow 2.5.x . If you are using TensorFlow version 2.5, you will receive the following warning:  tensorflow\python\keras\engine\sequential.py:455: UserWarning: model.predict_classes() is deprecated and will be removed after 2021-01-01. Please use instead:* np.argmax(model.predict(x), axis=-1), if your model does multi-class classification (e.g. if it uses a softmax last-layer activation).* (model.predict(x) > 0.5).astype("int32"), if your model does binary classification (e.g. if it uses a sigmoid last-layer activation).
Question: I have been investigating the use of Poetry to publish Python projects. I wanted to test the publishing process using a trivial project similar to the Python Packaging Authority tutorial. Since this is a trivial project, I want to publish it to the test instance of pypi rather than the real instance. Test.pypi requires a token to publish, but I can't figure out how to make Poetry use my test pypi token. All the documentation I can find uses HTTP basic authentication for test-pypi which no longer works. I added the repository using this command: poetry config.repositories.test-pypi https://test.pypi.org  I have tried creating tokens using both the following commands: poetry config pypi-token.test-pypi my-token  poetry config test-pypi-token.test-pypi my-token  I don't find a good explanation of the syntax for adding tokens in the poetry documentation, so any help will be appreciated.

Answer: I've successfully used tokens and poetry to upload to PyPI and TestPyPI. I believe you just need to change the TestPyPI URL you are configuring by appending /legacy/: poetry config repositories.test-pypi https://test.pypi.org/legacy/  You can then create your token as you were doing previously: poetry config pypi-token.test-pypi <your-token>  https://test.pypi.org/legacy/ is the API endpoint for uploading packages. It's a bit hidden in the documentation but it is mentioned here that that is the URL you should use. Also note that the name succeeding the period in repositories. and pypi-token. is what needs to match which is why we have specified: repositories.test-pypi and pypi-token.test-pypi
Question: I'm running a little python project to collect data. It's being triggered by a scheduled GitHub Action script (every midnight). As part of expanding the project I've added the pycaret library to the project. So currently installing the requirements for the project is taking about 15 minutes, plus running the python project is another 10 minutes. But the interesting part is that now the action/job is being killed with: /home/runner/work/_temp/bad86621-8542-4ea5-ae93-6f59b7ee2463.sh: line 1: 4382 Killed python main.py Error: Process completed with exit code 137.  Now i'vew tried looking up the reason for the process being killed by i have found nothing in GitHub Actions i'm running the job on ubuntu-latest machine in GitHub actions. i've set the job timeout to 60 minutes , so i don't think that is the issue.

Answer: Error 137 indicates that the container (runner/build agent) that builds your project received SIGKILL and terminated. It can be initiated manually or by the host machine when the runner exceeds its allocated memory limit. In your case, since it is initiated by Github itself, then it is generally due to being out of memory. (P.S. This is a very late answer, but it may help some folks)
Question: I have the following steps in a github action: steps: - name: Check out repository code uses: actions/checkout@v2 - name: Cache dependencies id: pip-cache uses: actions/cache@v2 with: path: ~.cache/pip key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }} restore-keys: | ${{ runner.os }}-pip- - name: Install dependencies if: steps.pip-cache.outputs.cache-hit != 'true' run: pip install -r requirements.txt - name: run mypy run: mypy .  The caching works fine, but when a cache hit occurs, and I try to run mypy, it fails with: Run mypy . /home/runner/work/_temp/9887df5b-d5cc-46d7-90e1-b884d8c49272.sh: line 1: mypy: command not found Error: Process completed with exit code 127.  The whole point of caching dependencies is so I don't have to install them every time I run the workflow. How do I use the cached dependencies?

Answer: You're only caching source tarballs and binary wheels downloaded by pip. You're not caching:  Installed Python packages (i.e., the site-packages/ subdirectory of the active Python interpreter). Installed entry points (i.e., executable commands residing in the current ${PATH}).  That isn't necessarily a bad thing. Merely downloading assets tends to consume a disproportionate share of scarce GitHub Actions (GA) minutes; caching assets trivially alleviates that issue. In other words, remove the if: steps.pip-cache.outputs.cache-hit != 'true' line to restore your GitHub Actions (GA) workflow to sanity. But... I Want to Cache Installed Packages! Challenge accepted. This is feasible – albeit more fragile. I'd advise just caching pip downloads unless you've profiled the pip install command to be a significant installation bottleneck. Let's assume that you still want to do this. In this case, something resembling the following snippet should get you where you want to go:  - uses: 'actions/setup-python@v2' with: # CAUTION: Replace this hardcoded "3.7" string with the # major and minor version of your desired Python interpreter. python-version: "3.7" - uses: 'actions/cache@v2' id: cache with: # CAUTION: Replace this hardcoded "python3.7" dirname with # the dirname providing your desired Python interpreter. path: ${{ env.pythonLocation }}/lib/python3.7/site-packages/* key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }} restore-keys: | ${{ runner.os }}-pip- ${{ runner.os }}-  As noted, you'll want to manually replace the hardcoded 3.7 and python3.7 substrings above with something specific to your use case. That's one of several reasons why this is dangerously fragile. Another is that the ${{ env.pythonLocation }} environment variable set by the setup-python GitHub Action has been infamously undocumented for several years. </gulp> (env.pythonLocation documentation was added from v4.0.0.) In theory, adding the above directly under your existing uses: actions/cache@v2 list item should suffice. Good luck and Godspeed as you travel into the terrifying unknown.
Question: Intellisense works fine on importing phrase  But when it comes with chaining method, it shows different suggestions  Python & Pylance extensions are installed.

Answer: From this issue on github try adding this to the bottom of your tensorflow/__init__.py (in .venv/Lib/site-packages/tensorflow for me) # Explicitly import lazy-loaded modules to support autocompletion. # pylint: disable=g-import-not-at-top if _typing.TYPE_CHECKING: from tensorflow_estimator.python.estimator.api._v2 import estimator as estimator from keras.api._v2 import keras from keras.api._v2.keras import losses from keras.api._v2.keras import metrics from keras.api._v2.keras import optimizers from keras.api._v2.keras import initializers # pylint: enable=g-import-not-at-top  The problem is because keras is a special class that enables lazy loading and not a normal module. Edit: With updates to tf, vscode, or something else I'm not having this issue and don't need to use the above fix anymore. I just have to use keras = tf.keras instead of from tensorflow import keras and I have Intellisense working now.
Question: The following code: df = df.drop('market', 1)  generates the warning:  FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only  market is the column we want to drop, and we pass the 1 as a second parameter for axis (0 for index, 1 for columns, so we pass 1). How can we change this line of code now so that it is not a problem in the future version of pandas / to resolve the warning message now?

Answer: From the documentation, pandas.DataFrame.drop has the following parameters: Parameters   labels: single label or list-like Index or column labels to drop.  axis: {0 or ‘index’, 1 or ‘columns’}, default 0 Whether to drop labels from the index (0 or ‘index’) or columns (1 or ‘columns’).  index: single label or list-like Alternative to specifying axis (labels, axis=0 is equivalent to index=labels).  columns: single label or list-like Alternative to specifying axis (labels, axis=1 is equivalent to columns=labels).  level: int or level name, optional For MultiIndex, level from which the labels will be removed.  inplace: bool, default False If False, return a copy. Otherwise, do operation inplace and return None.  errors: {‘ignore’, ‘raise’}, default ‘raise’ If ‘ignore’, suppress error and only existing labels are dropped.    Moving forward, only labels (the first parameter) can be positional.  So, for this example, the drop code should be as follows: df = df.drop('market', axis=1)  or (more legibly) with columns: df = df.drop(columns='market')
Question: I am running a Flask application in Apache using mod_wsgi. When I try to import numpy, I get the following warning:  /usr/local/lib/python3.8/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve.  Should I do anything to address this warning?

Answer: Following the information here, you can eliminate the warning (and prevent potential problems) by adding WSGIApplicationGroup %{GLOBAL}  to your httpd.conf.
Question: I was programming a game using Python and a sound effect needed to be played, so I used the playsound module: from playsound import playsound playsound("Typing.wav", False)  And when I attempted the run the program this error was returned: Error 263 for command: open Typing.wav The specified device is not open or is not recognized by MCI.  I did some research and some sources indicated that it was an issue with my sound drivers. I updated & reinstalled it, but the issue persists. Is there any way to solve this?

Answer: I faced this problem too firstly as mentioned in the previous comments I downgraded my python version from 3.10 to 3.7 and yet the problem persisted. So what actually worked is that the recent versions of playsound are giving such errors in order to fix this run the following commands in cmd as admin pip uninstall playsound pip install playsound==1.2.2 and this should do the work. just in case that doesn't work try degrading your python version to 3.7 and run these commands and that should be good.
Question: I observed the following behavior with DataFrame.to_json: >>> df = pd.DataFrame([[eval(f'1.12345e-{i}') for i in range(8, 20)]]) >>> df 0 1 2 3 4 5 6 7 8 9 10 11 0 1.123450e-08 1.123450e-09 1.123450e-10 1.123450e-11 1.123450e-12 1.123450e-13 1.123450e-14 1.123450e-15 1.123450e-16 1.123450e-17 1.123450e-18 1.123450e-19 >>> print(df.to_json(indent=2, orient='index')) { "0":{ "0":0.0000000112, "1":0.0000000011, "2":0.0000000001, "3":0.0, "4":0.0, "5":0.0, "6":0.0, "7":0.0, "8":1.12345e-16, "9":1.12345e-17, "10":1.12345e-18, "11":1.12345e-19 } }  So all numbers down to 1e-16 seem to be rounded to 10 decimal places (in agreement with the default value for double_precision) but all smaller values are represented exactly. Why is this the case and how can I turn off decimal rounding for the larger values too (i.e. using scientific notation instead)?  >>> pd.__version__ '1.3.1'   For reference, the standard library's json module doesn't do this: >>> import json >>> print(json.dumps([eval(f'1.12345e-{i}') for i in range(8, 20)], indent=2)) [ 1.12345e-08, 1.12345e-09, 1.12345e-10, 1.12345e-11, 1.12345e-12, 1.12345e-13, 1.12345e-14, 1.12345e-15, 1.12345e-16, 1.12345e-17, 1.12345e-18, 1.12345e-19 ] 

Answer: I'm not sure on achieving this with pd.DataFrame.to_json, but we can use pd.DataFrame.to_dict, json, and pd.read_json to achieve a full precision json representation from a pandas dataframe. json_df = json.dumps(df.to_dict('index'), indent=2) >>> print(json_df) { "0": { "0": 1.12345e-08, "1": 1.12345e-09, "2": 1.12345e-10, "3": 1.12345e-11, "4": 1.12345e-12, "5": 1.12345e-13, "6": 1.12345e-14, "7": 1.12345e-15, "8": 1.12345e-16, "9": 1.12345e-17, "10": 1.12345e-18, "11": 1.12345e-19 } }  To read it back in, we can then do: >>> pd.read_json(json_df, orient='index') 0 1 2 ... 9 10 11 0 1.123450e-08 1.123450e-09 1.123450e-10 ... 1.123450e-17 1.123450e-18 1.123450e-19 [1 rows x 12 columns]
Question: Hi I am getting the following warning (A squiggly line underneath imports), import "numpy" could not be resolved Pylance(reportMissingModuleSource). There is no issues with executing the code - works fine, just the warning (squiggly line). In the following github page, it states to change Settings.JSON with following line "python.analysis.extraPaths": ["./sources"].  https://github.com/microsoft/pylance-release/blob/main/TROUBLESHOOTING.md#unresolved-import-warnings  However this didn't work. I also tried adding the path to the current directory followed by "sources" as show in the image. But it didnt work either. I am opening vs code from this entry point /home/imantha/workspace/python using bash with code . command. Could anyone know how I add the correct path. 

Answer: If I understand your problem correctly, your python environment is properly set (for you are able to run your code) but your IDE (Vs code) points import errors. That is probably because your IDE does not know which python environment use for your current project (which seems to live somewhere in /home/imantha/workspace/python). You need to set it to get rid of this warning https://code.visualstudio.com/docs/python/environments
Question: How can one install a .whl (python library) from a private github repo? I have setup a personal access token and can install the library if its not a .whl by using the following command pip install git+https://{token}@github.com/{org_name}/{repo_name}.git  However if there is a .whl in the repo and I want to install from that using: pip install git+https://{token}@github.com/{org_name}/{repo_name}/blob/master/{name.whl}  Then I get the following error: TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType  I am stumped! You can pip install {name.whl} if the file is locally but not from the private github repo. Quesiton: How to pip install name.whl on a private github repo?

Answer: You should be able to do pip install https://{token}@raw.githubusercontent.com/{user}/{repo}/master/{name.whl}