eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have unit tests that require test data. This test data is downloaded and has decently large filesize. @pytest.fixture def large_data_file(): large_data = download_some_data('some_token') return large_data # Perform tests with input data def test_foo(large_data_file): pass def test_bar(large_data_file): pass def test_baz(large_data_file): pass # ... and so on  I do not want to download this data more than once. It should only be downloaded once, and passed to all the tests that require it. Does pytest call on large_data_file once and use it for every unit test that uses that fixture, or does it call the large_data_file each time? In unittest, you would simply download the data for all the tests once in the setUpClass method. I would rather not just have a global large_data_file = download_some_data('some_token') in this py script. I would like to know how to handle this use-case with Pytest.

Answer:  Does pytest call on large_data_file once and use it for every unit test that uses that fixture, or does it call the large_data_file each time?  It depends on the fixture scope. The default scope is function, so in your example large_data_file will be evaluated three times. If you broaden the scope, e.g. @pytest.fixture(scope="session") def large_data_file(): ...  the fixture will be evaluated once per test session and the result will be cached and reused in all dependent tests. Check out the section Scope: sharing fixtures across classes, modules, packages or session in pytest docs for more details.
Question: I want to add a fill colour between the black and blue line on my Plotly chart. I am aware this can be accomplished already with Plotly but I am not sure how to fill the chart with two colours based on conditions. The chart with the blue background is my Plotly chart. I want to make it look like the chart with the white background. (Ignore the red and green bars on the white chart) The conditions I want it to pass is: Fill the area between the two lines GREEN, if the black line is above the blue line. Fill the area between the two lines RED, if the black line is below the blue line. How can this be done with Plotly? If this is not possible with Plotly can it be accomplished with other graphing tools that work with Python.

Answer: For a number of reasons (that I'm willing to explain further if you're interested) the best approach seems to be to add two traces to a go.Figure() object for each time your averages cross eachother, and then define the fill using fill='tonexty' for the second trace using: for df in dfs: fig.add_traces(go.Scatter(x=df.index, y = df.ma1, line = dict(color='rgba(0,0,0,0)'))) fig.add_traces(go.Scatter(x=df.index, y = df.ma2, line = dict(color='rgba(0,0,0,0)'), fill='tonexty', fillcolor = fillcol(df['label'].iloc[0])))  fillcol is a simple custom function described in the full snippet below. And I've used the approach described in How to split a dataframe each time a string value changes in a column? to produce the necessary splits in the dataframe each time your averages cross eachother. Plot  Complete code: import plotly.graph_objects as go import numpy as np import pandas as pd from datetime import datetime pd.options.plotting.backend = "plotly" # sample data df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/finance-charts-apple.csv') df.index = df.Date df = df[['AAPL.Close', 'mavg']] df['mavg2'] = df['AAPL.Close'].rolling(window=50).mean() df.columns = ['y', 'ma1', 'ma2'] df=df.tail(250).dropna() df1 = df.copy() # split data into chunks where averages cross each other df['label'] = np.where(df['ma1']>df['ma2'], 1, 0) df['group'] = df['label'].ne(df['label'].shift()).cumsum() df = df.groupby('group') dfs = [] for name, data in df: dfs.append(data) # custom function to set fill color def fillcol(label): if label >= 1: return 'rgba(0,250,0,0.4)' else: return 'rgba(250,0,0,0.4)' fig = go.Figure() for df in dfs: fig.add_traces(go.Scatter(x=df.index, y = df.ma1, line = dict(color='rgba(0,0,0,0)'))) fig.add_traces(go.Scatter(x=df.index, y = df.ma2, line = dict(color='rgba(0,0,0,0)'), fill='tonexty', fillcolor = fillcol(df['label'].iloc[0]))) # include averages fig.add_traces(go.Scatter(x=df1.index, y = df1.ma1, line = dict(color = 'blue', width=1))) fig.add_traces(go.Scatter(x=df1.index, y = df1.ma2, line = dict(color = 'red', width=1))) # include main time-series fig.add_traces(go.Scatter(x=df1.index, y = df1.y, line = dict(color = 'black', width=2))) fig.update_layout(showlegend=False) fig.show()
Question: I am using sublime to code python scripts. The following code is for selenium in python to install the driver automatically by using the webdriver_manager package # pip install webdriver-manager from selenium import webdriver from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager from selenium.webdriver.common.by import By driver = webdriver.Chrome(ChromeDriverManager().install()) driver.maximize_window() #s=Service(path) #driver=webdriver.Chrome(service=s) driver.get('https://www.google.com') driver.find_element(By.NAME, 'q').send_keys('Yasser Khalil')  The code works fine but I got a warning like that Demo.py:7: DeprecationWarning: executable_path has been deprecated, please pass in a Service object driver = webdriver.Chrome(ChromeDriverManager().install())  How to fix such a bug?

Answer: This error message... DeprecationWarning: executable_path has been deprecated, please pass in a Service object  ...implies that the key executable_path will be deprecated in the upcoming releases. This change is inline with the Selenium 4.0 Beta 1 changelog which mentions:  Deprecate all but Options and Service arguments in driver instantiation. (#9125,#9128)   Solution With selenium4 as the key executable_path is deprecated you have to use an instance of the Service() class along with ChromeDriverManager().install() command as discussed below. Pre-requisites Ensure that:  Selenium is upgraded to v4.0.0 pip3 install -U selenium   Webdriver Manager for Python is installed pip3 install webdriver-manager     You can find a detailed discussion on installing Webdriver Manager for Python in ModuleNotFoundError: No module named 'webdriver_manager' error even after installing webdrivermanager  Selenium v4 compatible Code Block from selenium import webdriver from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager driver = webdriver.Chrome(service=Service(ChromeDriverManager().install())) driver.get("https://www.google.com")  Console Output: [WDM] - ====== WebDriver manager ====== [WDM] - Current google-chrome version is 96.0.4664 [WDM] - Get LATEST driver version for 96.0.4664 [WDM] - Driver [C:\Users\Admin\.wdm\drivers\chromedriver\win32\96.0.4664.45\chromedriver.exe] found in cache   You can find a detailed discussion on installing Webdriver Manager for Python in Selenium ChromeDriver issue using Webdriver Manager for Python   Incase you want to pass the Options() object you can use: from selenium import webdriver from selenium.webdriver.chrome.options import Options from selenium.webdriver.chrome.service import Service from webdriver_manager.chrome import ChromeDriverManager options = Options() options.add_argument("start-maximized") driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options) driver.get("https://www.google.com")   TL; DR You can find the relevant Bug Report/Pull Request in:  Bug Report: deprecate all but Options and Service arguments in driver instantiation Pull Request: deprecate all but Options and Service arguments in driver instantiation
Question: I want to fine-tune BERT on texts that are related to a specific domain (in my case related to engineering). The training should be unsupervised since I don't have any labels or anything. Is this possible?

Answer: What you in fact want to is continue pre-training BERT on text from your specific domain. What you do in this case is to continue training the model as masked language model, but on your domain-specific data. You can use the run_mlm.py script from the Huggingface's Transformers.
Question: def get_db(): db = SessionLocal() try: return db finally: db.close()  I got this code snipped to get Sessionlocal in fastapi with Sqlalchemy. Well, when I used return instead of Yield. My code still works. Then, I do not understand the reason of using Yield. Can someone help me?

Answer:  Well, when I used return instead of Yield. My code still works. Then, I do not understand the reason of using Yield.  It's a great question, the answer is, yes there's a reason to use yield instead of return. SQLAlchemy has a Connection Pooling mechanism by default. That means with yield you are creating a single session for each request. When you use return you are using a single database connection for all your app. Isn't it clear? Let's visualize it to make things more interesting.  So what do we have in this example?  A connection pool that holds 5 different connections. 2 endpoint 3 incoming requests.  When you use yield it would look something like this underneath because it goes to an endpoint, asks something to the database and yield creates a new Session object every time. It provides a transactional scope around a series of operations, but using return there instead of yield will just return that session object.
Question: I'm having a problem with adding a logarithmic X-axis to my plot. I want to show results based on the sample size with methods A, B and C. My dataframe result:  A B C 15 0.733333 0.613333 0.733333 30 0.716667 0.693333 0.766667 59 0.733684 0.678485 0.745763 118 0.796667 0.726087 0.779661 236 0.817862 0.788333 0.838983 470 0.832125 0.814468 0.836170  What I'm trying to make work: sample_count = np.around(np.logspace(math.log10(15),math.log10(470),6)) sample_count = sample_count.astype(int) sns.set_style('whitegrid') g_results=sns.lineplot(data=results,dashes=0,markers=['o','o','o']) g_results.set(xticks=sample_count) g_results.set(xscale='log')  However the result is not what I exactly want, as the ticks are completely gone:  Without the last xscale line it looks like this, which is the linear scale of course, but this time with the correct ticks:  What I want to achieve is something like this:  How do I get my desired output?

Answer: First set the scale for the x-axis to logarithmic and then set xticks and labels as you want. sns.set_style('whitegrid') g_results=sns.lineplot(data=results,dashes=0,markers=['o','o','o']) g_results.set(xscale='log') g_results.set(xticks=sample_count) g_results.set(xticklabels=sample_count)  This gives you this result:  Note that I'm using sample_count as defined with the logarithmic scale: sample_count = np.around(np.logspace(math.log10(15),math.log10(470),6))
Question: I am trying to upscale image (performing super resolution) using OpenCV, but I am getting this error that module 'cv2.cv2' has no attribute 'dnn_superres'. Any help would be greatly appreciated. I am using 4.4.0.44 OpenCV version. Here is the code section. import cv2 sr = cv2.dnn_superres.DnnSuperResImpl_create() sr.readModel(args["model"]) sr.setModel(modelName, modelScale) 

Answer: In case you are using python3, you need to download opencv with pip3 First uninstall opencv:  pip uninstall opencv-python pip uninstall opencv-contrib-python  Then install latest version of opencv with pip3: pip3 install opencv-contrib-python
Question: I need python newest version in raspberry pi. I tried apt install python3 3.8 apt install python3 but this didnot work. And I also needed to update my raspberry pi python IDLE

Answer: First update the Raspbian. sudo apt-get update  Then install the prerequisites that will make any further installation of Python and/or packages much smoother. sudo apt-get install -y build-essential tk-dev libncurses5-dev libncursesw5-dev libreadline6-dev libdb5.3-dev libgdbm-dev libsqlite3-dev libssl-dev libbz2-dev libexpat1-dev liblzma-dev zlib1g-dev libffi-dev  And then install Python, maybe by downloading a compressed file? example 1 : wget https://www.python.org/ftp/python/3.8.0/Python-3.8.0.tgz  Extract the folder : sudo tar zxf Python-3.8.0.tgz  Move into the folder : cd Python-3.8.0  Initial configuration : sudo ./configure --enable-optimizations  Run the makefile inside the folder with the mentioned parameters : sudo make -j 4  Run again the makefile this time installing directly the package : sudo make altinstall  Maybe You already did it but You don't know how to setup the new version as a default version of the system? Check first that it has been installed : python3.8 -V  Send a strong command to .bashrc telling him who (which version) is in charge of Python echo "alias python=/usr/local/bin/python3.8" >> ~/.bashrc  Again! Tell him because .bashrc has to understand! I am joking - You have to source the file so the changes can be applied immediately : source ~/.bashrc  And then check that Your system changed the default version of Python to Python 3.8 python -V  The failure depends on many factors : what dependencies are installed, what are the packages added to the source_list.d, some inconvenient coming up during the installation. All may give you more information than you think, just read carefully. Hope it helped.
Question: I recently started using Jupyter Notebooks on vscode but i've notices that code autocompletion doesn't work properly. If i create a regular .py file everything works correctly as you can see. It shows function signature and docstring. In both core python language and extern modules.   But if i try the same in a .ipynb file it completely ignores autocompletion for print()  And what confuses me too if that for example it shows me np.sum() docstring but it doesn't show me any np.concatenate() information gamong any other np.functions or other modules [  Just in case im using Vscode and an conda enviroment as my python interpreter. Here is my settings.json file: { "python.dataScience.jupyterServerURI": "local", "python.pythonPath": "C:\\Users\\myUser\\anaconda3\\envs\\myEnv\\python.exe" } 

Answer: According to your description, the reason for this situation is that different language services provide different functions such as automatic completion and prompts. For the "print()" and "np.concatenate()" you mentioned, it is recommended that you use the "Pylance" extension, which provides excellent language services and auto-complete functions. Please add the following settings in settings.json: "python.languageServer": "Pylance",    Update: Starting from November 2020, the function of Jupyter notebook in VSCode is provided by the extension "Jupyter", which uses the "IntelliSense" provided by the extension "Jupyter". And in "VSCode-insider" Jupyter notebook has better "IntelliSense":  Github link: Hover Documentation Pop up does not work after VSCode 1.52 update.
Question: I am upgrading a Django app from 2.2.7 to 3.1.3. The app uses Postgres 12 & psycopg2 2.8.6. I followed the instructions and changed all my django.contrib.postgres.fields.JSONField references to django.db.models.JSONField, and made and ran the migrations. This produced no changes to my schema (which is good.) However, when I execute a raw query the data for those jsonb columns is returned as text, or converted to text, at some point. I don't see this issue when querying the models directly using Model.objects.get(...). import os, django os.environ.setdefault("DJANGO_SETTINGS_MODULE", "big_old_project.settings") django.setup() with connection.cursor() as c: c.execute("select name, data from tbl where name=%s", ("rex",)) print(c.description) for row in c.fetchall(): for col in row: print(f"{type(col)} => {col!r}") (Column(name='name', type_code=1043), Column(name='data', type_code=3802)) <class 'str'> => 'rex' <class 'str'> => '{"toy": "bone"}'  [edit] Using a raw connection gives the expected results: conn = psycopg2.connect("dbname=db user=x password=z") with conn.cursor() as c: ... <class 'str'> => 'rex' <class 'dict'> => {'toy': 'bone'}  Trying the old trick of "registering" the adapter doesn't work, and shouldn't be needed anyway. import psycopg2.extras psycopg2.extras.register_json(oid=3802, array_oid=3807, globally=True)  This app has a lot of history, so maybe something is stepping on psycopg2's toes? I can't find anything so far, and have commented out everything that seems tangentially related. Going through the release notes didn't help. I do use other postgres fields so I can't delete all references to contrib.postgres.fields from my models. Any ideas as to why this is happening would be greatly appreciated.

Answer: To add to @Andrew Backer's helpful answer, this is apparently intentional. From the 3.1.1 release notes:  Fixed a QuerySet.order_by() crash on PostgreSQL when ordering and grouping by JSONField with a custom decoder (#31956). As a consequence, fetching a JSONField with raw SQL now returns a string instead of pre-loaded data. You will need to explicitly call json.loads() in such cases.  It's surprising to find an API-incompatible change as an aside in a bugfix release. For now I'll be adding json.loads() calls since, as already mentioned, there's no guarantee the ::json workaround doesn't break as well!
Question: My code looks something like this: from joblib import Parallel, delayed # prediction model - 10s of megabytes on disk LARGE_MODEL = load_model('path/to/model') file_paths = glob('path/to/files/*') def do_thing(file_path): pred = LARGE_MODEL.predict(load_image(file_path)) return pred Parallel(n_jobs=2)(delayed(do_thing)(fp) for fp in file_paths)  My question is whether LARGE_MODEL will be pickled/unpickled with each iteration of the loop. And if so, how can I make sure each worker caches it instead (if that's possible)?

Answer: TLDR  The parent process pickles large model once. That can be made more performant by ensuring large model is a numpy array backed to a memfile. Workers can load_temporary_memmap much faster than from disk.  Your job is parallelized and likely to be using joblibs._parallel_backends.LokyBackend. In joblib.parallel.Parallel.__call__, joblib tries to initialize the backend to use LokyBackend when n_jobs is set to a count greater than 1. LokyBackend uses a shared temporary folder for the same Parallel object. This is relevant for reducers that modify the default pickling behavior. Now, LokyBackend configures a MemmappingExecutor that shares this folder to the reducers. If you have numpy installed and your model is a clean numpy array, you are guaranteed to have it pickled once as a memmapped file using the ArrayMemmapForwardReducer and passed from parent to child processes. Otherwise it is pickled using the default pickling as a bytes object. You can know how your model is pickled in the parent process reading the debug logs from joblib. Each worker 'unpickles' large model so there is really no point in caching the large model there. You can only improve the source from where the pickled large model is loaded from in the workers by backing your models as a memory mapped file.
Question: I am connecting to a SQL Server database using SQLAlchemy (with the pymssql driver). import sqlalchemy conn_string = f'mssql+pymssql://{uid}:{pwd}@{instance}/?database={db};charset=utf8' sql = 'SELECT * FROM FAKETABLE;' engine = sqlalchemy.create_engine(conn_string) connection = engine.connect() result = connection.execute(sql) result.cursor.description  which results in: (('col_1', 1, None, None, None, None, None), ('col_2', 1, None, None, None, None, None), ('col_3', 4, None, None, None, None, None), ('col_4', 3, None, None, None, None, None), ('col_5', 3, None, None, None, None, None))  As per PEP 249 (cursor's .description attribute):  The first two items (name and type_code) are mandatory, the other five are optional and are set to None if no meaningful values can be provided.  I am assuming the integers (1, 1, 4, 3, 3) are column types. My two questions:  How to map these integers to data types (like char, integer, etc.)? Are these SQL data types? If no, is it possible to get the SQL data types?  FWIW, I get the same result when using raw_connection() instead of connect(). Came across three questions along similar lines (which do not answer this specific question). I need to use the connect() + execute() approach.  SQLAlchemy getting column data types of query results How to get columns' sql type queried by sqlalchemy Easy convert betwen SQLAlchemy column types and python data types? 

Answer:  If no, is it possible to get the SQL data types?  SQL Server function sys.dm_exec_describe_first_result_set  could be used to get SQL column's data type directly for provided query: SELECT column_ordinal, name, system_type_name, * FROM sys.dm_exec_describe_first_result_set('here goes query', NULL, 0) ;  In your example: sql = """SELECT column_ordinal, name, system_type_name FROM sys.dm_exec_describe_first_result_set('SELECT * FROM FAKETABLE', NULL, 0) ;"""  For: CREATE TABLE FAKETABLE(id INT, d DATE, country NVARCHAR(10)); SELECT column_ordinal, name, system_type_name FROM sys.dm_exec_describe_first_result_set('SELECT * FROM FAKETABLE', NULL, 0) ; +-----------------+----------+------------------+ | column_ordinal | name | system_type_name | +-----------------+----------+------------------+ | 1 | id | int | | 2 | d | date | | 3 | country | nvarchar(10) | +-----------------+----------+------------------+  db<>fiddle demo
Question: I am having issues with Pipenv. I run pipenv install --dev in order to install some dependencies from a Pipfile within my project. Upon running this command, Pipenv generates an MD5 hash for a certain dependency. The error is saying that MD5 is not supported yet still generates it. I have not set any configurations on my local machine or in any configuration file. I cannot seem to pinpoint this issue. Any help is greatly appreciated. [pipenv.exceptions.InstallError]: pip: error: Allowed hash algorithms for --hash are sha256, sha384, sha512. Pipfile.lock { "_meta": { "hash": { "sha256": "7e0f1d75f7df19f9500f55bd2f1da163cb4a8c7f485aab61c521d70e3865a507" }, "pipfile-spec": 6, "requires": { "python_version": "3.6" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": { "certain-dependency": { "hashes": [ "md5:8faf2e4ff85c34b5d0c000c017f81f52", "md5:1508a7f05b17d292f7890b8c58a451cf", ], "version": "==11.10.20" } } } 

Answer: Try clearing your pipenv cache:  Make sure your dependencies actually do resolve. If you’re confident they are, you may need to clear your resolver cache. Run the following command: pipenv lock --clear  and try again. If this does not work, try manually deleting the whole cache directory. It is usually one of the following locations:  ~/Library/Caches/pipenv (macOS) %LOCALAPPDATA%\pipenv\pipenv\Cache (Windows) ~/.cache/pipenv (other operating systems)   While the current release of pipenv only accepts sha256 hashes, it loads package URLs from a cache and writes those cached URLs' hashes to Pipfile.lock. If those cached hashes are md5 hashes from previously-installed packages, pipenv uses those values as-is without verifying that they are sha256/FAVORITE_HASH. Clearing the cache and re-locking will cause pipenv to cache miss and re-fetch package URLs which end in sha256 hashes, and write them to Pipfile.lock as you'd hope, and prevent you from running into the issue again.