eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: How to use tqdm for data_loader ? is this the correct way? for i,j in enumerate(data_loader,total = 100): pass 

Answer: You need to wrap the iterable with tqdm, as their documentation clearly says:  Instantly make your loops show a smart progress meter - just wrap any iterable with tqdm(iterable), and you’re done!  If you're enumerating over an iterable, you can do something like the following. Sleep is only for visualizing it. from tqdm import tqdm from time import sleep data_loader = list(range(1000)) for i, j in enumerate(tqdm(data_loader)): sleep(0.01)
Question: I would love to use a schema that looks something like the following in FastAPI: from __future__ import annotations from typing import List from pydantic import BaseModel class Project(BaseModel): members: List[User] class User(BaseModel): projects: List[Project] Project.update_forward_refs()  but in order to keep my project structure clean, I would ofc. like to define these in separate files. How could I do this without creating a circular reference? With the code above the schema generation in FastAPI works fine, I just dont know how to separate it out into separate files. In a later step I would then instead of using attributes use @propertys to define the getters for these objects in subclasses of them. But for the OpenAPI doc generation, I need this combined - I think.

Answer: There are three cases when circular dependency may work in Python:  Top of module: import package.module Bottom of module: from package.module import attribute Top of function: works both  In your situation, the second case "bottom of module" will help. Because you need to use update_forward_refs function to resolve pydantic postponed annotations like this: # project.py from typing import List from pydantic import BaseModel class Project(BaseModel): members: "List[User]" from user import User Project.update_forward_refs()  # user.py from typing import List from pydantic import BaseModel class User(BaseModel): projects: "List[Project]" from project import Project User.update_forward_refs()  Nonetheless, I would strongly discourage you from intentionally introducing circular dependencies
Question: I need to replace multiple words in a html document. Atm I am doing this by calling replace_with once for each replacement. Calling replace_with twice on a NavigableString leads to a ValueError (see example below) cause the replaced element is no longer in the tree. Minimal example #!/usr/bin/env python3 from bs4 import BeautifulSoup import re def test1(): html = \ ''' Identify ''' soup = BeautifulSoup(html,features="html.parser") for txt in soup.findAll(text=True): if re.search('identify',txt,re.I) and txt.parent.name != 'a': newtext = re.sub('identify', '<a href="test.html"> test </a>', txt.lower()) txt.replace_with(BeautifulSoup(newtext, features="html.parser")) txt.replace_with(BeautifulSoup(newtext, features="html.parser")) # I called it twice here to make the code as small as possible. # Usually it would be a different newtext .. # which was created using the replaced txt looking for a different word to replace. return soup print(test1())  Expected Result: The txt is == newstring  Result: ValueError: Cannot replace one element with another when the element to be replaced is not part of the tree.  An easy solution would be just to tinker around with the newstring and only replacing all at once in the end, but I would like to understand the current phenomenon.

Answer: The first txt.replace_with(...) removes NavigableString (here stored in variable txt) from the document tree (doc). This effectively sets txt.parent to None The second txt.replace_with(...) looks at parent property, finds None (because txt is already removed from tree) and throws an ValueError. As you said at the end of your question, one the solution can be to use .replace_with() only once: import re from bs4 import BeautifulSoup def test1(): html = \ ''' word1 word2 word3 word4 ''' soup = BeautifulSoup(html,features="html.parser") to_delete = [] for txt in soup.findAll(text=True): if re.search('word1', txt, flags=re.I) and txt.parent.name != 'a': newtext = re.sub('word1', '<a href="test.html"> test1 </a>', txt.lower()) # ...some computations newtext = re.sub('word3', '<a href="test.html"> test2 </a>', newtext) # ...some more computations # and at the end, replce txt only once: txt.replace_with(BeautifulSoup(newtext, features="html.parser")) return soup print(test1())  Prints: <a href="test.html"> test1 </a> word2 <a href="test.html"> test2 </a> word4
Question: I've used many video downloaders before: atube catcher, 4k downloader, jDownloader, and currently using youtube-dl. I can't download videos, this for example, while still keeping their online chapters intact, like part1 is "intro" lasting from 00:00 to 00:45 and so on. So far I tried these parameters with youtube-dl Filesystem --write-annotations --write-description --write-info-json Thumbnail images --write-all-thumbnails Video format -f 'bestvideo[height<=720]+bestaudio/best[height<=720]/worst' --merge-output-format mp4 Post-processing --add-metadata --embed-subs --embed-thumbnail  Also tried requesting the mkv video format (thought maybe it was built into it) didn't help tho.  I know these options don't really say anything about sections but I'm trying to get as much metadata as I can

Answer: The information you want is called chapters in the youtube-dl info JSON. There is a recent open pull request for youtube-dl that fixes a problem with this information. In the current release of youtube-dl, if you use the ---write-info-json or --dump-json you will see that the chapters information is null ("chapters": null). You can use the code in the fork repository to be able to obtain the information you want. Follow these steps:  Clone this repository: git clone https://github.com/gschizas/youtube-dl.git   Change to the repository directory: cd youtube-dl/   Checkout the pull request branch: git checkout bugfix/youtube/chapters-fix-extractor   Run youtube-dl from the current location: python -m youtube_dl --write-info-json https://youtu.be/LnO42jxJaC4    You will see information like this in the info JSON: "chapters": [ { "start_time": 0.0, "end_time": 46.0, "title": "Intro" }, { "start_time": 46.0, "end_time": 72.0, "title": "QOTD" }, ... ]  Hopefully the fix will be accepted into the youtube-dl repository and included in future releases, so there will be no need to clone any repository.
Question: I have a Pandas dataframe that I'm writing out to Snowflake using SQLAlchemy engine and the to_sql function. It works fine, but I have to use the chunksize option because of some Snowflake limit. This is also fine for smaller dataframes. However, some dataframes are 500k+ rows, and at a 15k records per chunk, it takes forever to complete writing to Snowflake. I did some research and came across the pd_writer method provided by Snowflake, which apparently loads the dataframe much faster. My Python script does complete faster and I see it creates a table with all the right columns and the right row count, but every single column's value in every single row is NULL. I thought it was a NaN to NULL issue and tried everything possible to replace the NaNs with None, and while it does the replacement within the dataframe, by the time it gets to the table, everything becomes NULL. How can I use pd_writer to get these huge dataframes written properly into Snowflake? Are there any viable alternatives? EDIT: Following Chris' answer, I decided to try with the official example. Here's my code and the result set: import os import pandas as pd from snowflake.sqlalchemy import URL from sqlalchemy import create_engine from snowflake.connector.pandas_tools import write_pandas, pd_writer def create_db_engine(db_name, schema_name): return create_engine( URL( account=os.environ.get("DB_ACCOUNT"), user=os.environ.get("DB_USERNAME"), password=os.environ.get("DB_PASSWORD"), database=db_name, schema=schema_name, warehouse=os.environ.get("DB_WAREHOUSE"), role=os.environ.get("DB_ROLE"), ) ) def create_table(out_df, table_name, idx=False): engine = create_db_engine("dummy_db", "dummy_schema") connection = engine.connect() try: out_df.to_sql( table_name, connection, if_exists="append", index=idx, method=pd_writer ) except ConnectionError: print("Unable to connect to database!") finally: connection.close() engine.dispose() return True df = pd.DataFrame([("Mark", 10), ("Luke", 20)], columns=["name", "balance"]) print(df.head) create_table(df, "dummy_demo_table")  The code works fine with no hitches, but when I look at the table, which gets created, it's all NULLs. Again. 

Answer: Turns out, the documentation (arguably, Snowflake's weakest point) is out of sync with reality. This is the real issue: https://github.com/snowflakedb/snowflake-connector-python/issues/329. All it needs is a single character in the column name to be upper case and it works perfectly. My workaround is to simply do: df.columns = map(str.upper, df.columns) before invoking to_sql.
Question: I am writing a Python program with two threads. One displays a GUI and the other gets input from a scanner and saves data in an online database. The code works fine on my raspberry pi but if I try it on my MacBook Pro (Catalina 10.15.2), I get the above mentioned warning followed by my code crashing. Does anyone have an idea how to get it working or what causes the problem?

Answer: You likely use different Python versions. Your Python on your Raspberry PI still allows invalidating NSWindow drag regions outside the Main thread, while your Python in your MacBook Pro already stopped supporting this. You will likely need to refactor your code so that NSWindow drag regions will only be invalidated on the Main thread. You need to localize where NSWindow drag regions are invalidated and make sure that those happen in the Main thread. EDIT The asker explained that according to his/her findings, NSWindow drag regions only apply to Mac.
Question: I have a large DataFrame with the following columns: import pandas as pd x = pd.read_csv('age_year.csv') x.head() ID Year age 22445 1991 29925 1991 76165 1991 223725 1991 16.0 280165 1991  The Year column has values ranging from 1991 to 2017. Most ID have an age value in each Year, for example: x.loc[x['ID'] == 280165].to_clipboard(index = False) ID Year age 280165 1991 280165 1992 280165 1993 280165 1994 280165 1995 16.0 280165 1996 17.0 280165 1997 18.0 280165 1998 19.0 280165 1999 20.0 280165 2000 21.0 280165 2001 280165 2002 280165 2003 280165 2004 25.0 280165 2005 26.0 280165 2006 27.0 280165 2007 280165 2008 280165 2010 31.0 280165 2011 32.0 280165 2012 33.0 280165 2013 34.0 280165 2014 35.0 280165 2015 36.0 280165 2016 37.0 280165 2017 38.0  I want to fill the missing values in the age column for each unique ID based on their existing values. For example, for ID 280165 above, we know they are 29 in 2008, given that they are 31 in 2010 (28 in 2007, 24 in 2003 and so on). How should one fill in these missing age values for many unique ID for every year? I'm not sure how to do this in a uniform way across the entire DataFrame. The data used as the example in this question can be found here.

Answer: I think instead of trying to fill the values, find the year of birth instead. df["age"] = df["Year"] - (df["Year"]-df["age"]).mean()  Or general solution with more than 1 id: s = df.loc[df["age"].notnull()].groupby("ID").first() df["age"] = df["Year"]-df["ID"].map(s["Year"]-s["age"]) print (df) ID Year age 0 280165 1991 12.0 1 280165 1992 13.0 2 280165 1993 14.0 3 280165 1994 15.0 4 280165 1995 16.0 5 280165 1996 17.0 6 280165 1997 18.0 7 280165 1998 19.0 8 280165 1999 20.0 9 280165 2000 21.0 10 280165 2001 22.0 11 280165 2002 23.0 12 280165 2003 24.0 13 280165 2004 25.0 14 280165 2005 26.0 15 280165 2006 27.0 16 280165 2007 28.0 17 280165 2008 29.0 18 280165 2010 31.0 19 280165 2011 32.0 20 280165 2012 33.0 21 280165 2013 34.0 22 280165 2014 35.0 23 280165 2015 36.0 24 280165 2016 37.0 25 280165 2017 38.0
Question: https://plotly.com/python/line-and-scatter/ has many scatter plot examples, but not a single one showing you how to set all the points' colours within px.scatter: # x and y given as DataFrame columns import plotly.express as px df = px.data.iris() # iris is a pandas DataFrame fig = px.scatter(df, x="sepal_width", y="sepal_length") fig.show()  I've tried adding colour = 'red' etc doesn't work. These examples only show you how to colour by some other variable. In principle I could add another feature and set it all the same but that seems a bizzare way of accomplishing the task....

Answer: For that you may use the color_discrete_sequence argument. fig = px.scatter(df, x="sepal_width", y="sepal_length", color_discrete_sequence=['red'])  This argument is to use a custom color paletter for discrete color factors, but if you are not using any factor for color it will use the first element for all the points in the plot. More about discrete color palletes: https://plotly.com/python/discrete-color/
Question: When I execute this command: pip3 install opencv-python I get the following error:  Installing build dependencies ... error ERROR: Command errored out with exit status 1: command: /usr/bin/python3 /usr/lib/python3/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-z4c_sn6u/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel scikit-build cmake pip 'numpy==1.11.3; python_version=='"'"'3.5'"'"'' 'numpy==1.13.3; python_version=='"'"'3.6'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"'' cwd: None Complete output (22 lines): Ignoring numpy: markers 'python_version == "3.5"' don't match your environment Ignoring numpy: markers 'python_version == "3.6"' don't match your environment Ignoring numpy: markers 'python_version == "3.7"' don't match your environment Collecting setuptools Downloading setuptools-49.6.0-py3-none-any.whl (803 kB) Collecting wheel Downloading wheel-0.35.0-py2.py3-none-any.whl (24 kB) Collecting scikit-build Using cached scikit_build-0.11.1-py2.py3-none-any.whl (72 kB) Collecting cmake Using cached cmake-3.18.0.tar.gz (28 kB) ERROR: Command errored out with exit status 1: command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-95tsmt_u/cmake/setup.py'"'"'; __file__='"'"'/tmp/pip-install-95tsmt_u/cmake/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-95tsmt_u/cmake/pip-egg-info cwd: /tmp/pip-install-95tsmt_u/cmake/ Complete output (5 lines): Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-install-95tsmt_u/cmake/setup.py", line 7, in <module> from skbuild import setup ModuleNotFoundError: No module named 'skbuild' ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. ---------------------------------------- ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3/dist-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-z4c_sn6u/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel scikit-build cmake pip 'numpy==1.11.3; python_version=='"'"'3.5'"'"'' 'numpy==1.13.3; python_version=='"'"'3.6'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"'' Check the logs for full command output.  Same when I tried to install ecapture, I am using the latest python version

Answer: Try to upgrade your pip with pip install --upgrade pip  and then run the pip install opencv-python
Question: I have to do some cleanup with my (Mini)conda python packages to free some disk space, and I see people usually resort to the conda clean command to get this job done. Conda documentation says that it it's safe to do that, as it will only erase packages that "have never been used in any environment". I've never used conda's environments feature, though, and I don't know if I should be doing so. I just install my packages with conda install commands, run Jupyter Notebook and do all my work inside Jupyter. (I'm not a software engineer, just a regular dude using python and pandas to manage data.) Do I risk erase my favorite packages if i run conda clean? If I don't do any cleanup, will conda eventually engulf all of my disk space? Is there any command I can use to check how much disk space my python packages are taking.

Answer: At the very least, tarballs can be removed with no risk. Cleaning packages is done based on counting the number of hardlinks for the package. If there is only one hardlink, this implies the package is not referenced by any environment, and therefore can be removed. This will be the case for all packages that were previously in use but were superseded by other versions. The warning mainly applies to people who have environments across different disks but are using softlinks to limit redundancy. Unlike hardlinks, the file system does not keep track of softlink references, so there is no simple way to count the number of softlinks. Hence, when cleaning, those packages that are only connected to envs through softlinks (i.e., only report one hardlink) will be removed and thereby break the softlinking envs.
Question: I have this DataFrame lst = [['AAA',15,'BBB',20],['BBB',16,'AAA',12],['BBB',22,'CCC',15],['CCC',11,'AAA',31],['DDD',25,'EEE',35]] df = pd.DataFrame(lst,columns = ['name1','val1','name2','val2'])  which looks like this  name1 val1 name2 val2 0 AAA 15 BBB 20 1 BBB 16 AAA 12 2 BBB 22 CCC 15 3 CCC 11 AAA 31 4 DDD 25 EEE 35  I want this  name1 val1 name2 val2 0 AAA 31 BBB 22 1 BBB 22 AAA 31 2 BBB 22 CCC 15 3 CCC 15 AAA 31 4 DDD 25 EEE 35  replaced all values with the maximum value. we choose the maximum value from both val1 and val2 if i do this i will get the maximum from only val1 df["val1"] = df.groupby("name1")["val1"].transform("max") 

Answer: Try using pd.wide_to_long to melt that dataframe into a long form, then use groupby with transform to find the max value. Map that max value to 'name' and reshape back to four column (wide) dataframe: df_long = pd.wide_to_long(df.reset_index(), ['name','val'], 'index', j='num',sep='',suffix='\d+') mapper= df_long.groupby('name')['val'].max() df_long['val'] = df_long['name'].map(mapper) df_new = df_long.unstack() df_new.columns = [f'{i}{j}' for i,j in df_new.columns] df_new  Output:  name1 name2 val1 val2 index 0 AAA BBB 31 22 1 BBB AAA 22 31 2 BBB CCC 22 15 3 CCC AAA 15 31 4 DDD EEE 25 35
Question: I'm using windows 10 and I got rid of python 3.8 and installed 3.7 as the only python version on my system. When trying to install libraries using pip I now get the error: Fatal error in launcher: Unable to create process using '"c:\users\user\appdata\local\programs\python\python38-32\python.exe" "C:\Users\User\AppData\Local\Programs\Python\Python38-32\Scripts\pip.exe" install pygame_menu': The system cannot find the file specified.  when I checked in the console which -a pip I got: C:\Users\User>which -a pip /cygdrive/c/Users/User/AppData/Local/Programs/Python/Python38-32/Scripts/pip /cygdrive/c/Users/User/AppData/Local/Programs/Python/Python37/Scripts/pip  Now when I look for Python in my variable path it is alright... Anyways I can't figure out how to change the path of pip so the right one is selected... besides its pretty weird that ive uninstalled python and pip multiple times and it still gets it wrong every time during installation. Thanks

Answer: to fix do this :  check if you still have the python38-32 folder in your local variable list  Delete the "%userprofile%\AppData\Local\Programs\Python\Python38" folder  run pip from command line   if the problem still persists then type "environment variables" in the windows search box and add "%userprofile%\AppData\Local\Programs\Python\Python37" to your system variable named "Path" This should completely fix your problem if not  uninstall all the python files including py launcher reinstall python # when installing you must select ADD PYTHON to system environment variable
Question: Layout: workspace/ .vscode/launch.json main.py foo.py: def harr(): pass  launch.json: { "version": "0.2.0", "configurations": [ { "name": "Python: Module", "type": "python", "request": "launch", "cwd": "${workspaceFolder}/..", "module": "${workspaceFolderBasename}" } ] }  main.py and pylint error: from .foo import harr ^ Attempted relative import beyond top-level package - pylint(relative-beyond-top-level)  The program runs fine if not for the linter error. I have tried modifying sys.path in __init__.py and through pylintrc but the result is the same. Is there a way VSCode can run linting for a file and know at the same time that the file is part of a package? I've scanned through vscode linting and the pylint config docs but found nothing like what I'd expect. This should be so standard, what am I not getting? How is pylint supposed to work out relative imports if it has no idea of the top-level package?

Answer: Currently pylint cannot find modules accurately through relative imports, it will mess up the path, although the code can run. You could try the following two ways to solve it: 1.Add the following settings in the setting.json file.  "python.linting.pylintArgs": ["--disable=all", "--enable=F,E,unreachable,duplicate-key,unnecessary-semicolon,global-variable-not-assigned,unused-variable,binary-op-exception,bad-format-string,anomalous-backslash-in-string,bad-open-mode", "--disable=E0402", ],   (Since there is no issue with the code, we can turn off this type of pylint prompt.)  Since the relative import method will make pylint confusing, we can avoid such use. Use 'from foo import harr' instead of 'from .foo import harr'.   Reference: Default Pylint rules.
Question: Very simple question, but I can't find the answer to it. My IDE vs code (pylance) give me the warning/hint for a being possibly unbound. Why is this? How do I fix it? def f(): for i in range(4): a = 1 print(a) return a 

Answer: Because range(4) might be something empty (if you overwrite the built-in range), in which case the loop body will never run and a will not get assigned. Which is a problem when it's supposed to get returned. Maybe you can tell your IDE to ignore this and not show the warning. Or assign some meaningful default to a before the loop.
Question: I have a warning flash message in Flask that that appears before the user tries to submit a form based on background information about the user. If the user goes ahead and submits the form the way they were warned not to, they are prevented and see a second flash message. I'd like to clear the first flash message before the user sees the second. I've read the Flask documentation on flash messages and tried to google for the answer. I also read some of the Flask source code. No solution jumps out at me. Can anyone help me figure out how to clear a flash message?

Answer: This way you can clear the flash message as there is no predefined method to clear the flash message in Flask flash helpers. You can try the below code. It works for me and maybe useful to you. session.pop('_flashes', None)