eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a list with strings as below. candidates = ["Hello", "World", "HelloWorld", "Foo", "bar", "ar"]  And I want the list to be filtered as ["HelloWorld", "Foo", "Bar"], because others are substrings. I can do it like this, but don't think it's fast or elegant. def filter_not_substring(candidates): survive = [] for a in candidates: for b in candidates: if a == b: continue if a in b: break else: survive.append(a) return survive  Is there any fast way to do it?

Answer: How about: candidates = ["Hello", "World", "HelloWorld", "Foo", "bar", "ar"] result = [c for c in candidates if not any(c in o and len(o) > len(c) for o in candidates)] print(result)  Counter to what was suggested in the comments: from timeit import timeit def filter_not_substring(candidates): survive = [] for a in candidates: for b in candidates: if a == b: continue if a in b: break else: survive.append(a) return survive def filter_not_substring2a(candidates): return [c for c in candidates if not any(len(o) > len(c) and c in o for o in candidates)] def filter_not_substring2b(candidates): return [c for c in candidates if not any(c in o and len(o) > len(c) for o in candidates)] xs = ["Hello", "World", "HelloWorld", "Foo", "bar", "ar", "bar"] print(filter_not_substring(xs), filter_not_substring2a(xs), filter_not_substring2b(xs)) print(timeit(lambda: filter_not_substring(xs))) print(timeit(lambda: filter_not_substring2a(xs))) print(timeit(lambda: filter_not_substring2b(xs)))  Result: ['HelloWorld', 'Foo', 'bar', 'bar'] ['HelloWorld', 'Foo', 'bar', 'bar'] ['HelloWorld', 'Foo', 'bar', 'bar'] 1.5163685 4.6516653 3.8334089999999996  So, OP's solution is substantially faster, but filter_not_substring2b is still about 20% faster than 2a. So, putting the len comparison first doesn't save time. For any production scenario, OP's function is probably optimal - a way to speed it up might be to bring the whole problem into C, but I doubt that would show great gains, since the logic is pretty straightforward already and I'd expect Python to do a fairly good job of it as well. User @ming noted that OP's solution can be improved a bit: def filter_not_substring_b(candidates): survive = [] for a in candidates: for b in candidates: if a in b and a != b: break else: survive.append(a) return survive  This version of the function is somewhat faster, for me about 10-15% Finally, note that this is only just faster than 2b, even though it is very similar to the optimised solution by @ming, but almost 3x slower than their solution. It's unclear to me why that would be - if anyone has fairly certain thoughts on that, please share in the comments: def filter_not_substring_c(candidates): return [a for a in candidates if all(a not in b or a == b for b in candidates)]
Question: I need to execute a Python script inside the Ddev web docker image, but am having trouble figuring out what Debian python libraries are required to get Python binary with additional py package dependencies working.

Answer: Most of this is obsolete, because from DDEV v1.23.0 you can't easily get python 2 on DDEV at all, since it's been dropped from upstream. However, see @stasadev answer below for a great add-on that solves this. ddev add-on get stasadev/ddev-python2, see https://github.com/stasadev/ddev-python2. Python 2 on Ddev You really don’t want to be using Python 2 do you? (See caveats 1 & 2 below) Add the following in .ddev/config.yml: webimage_extra_packages: [python]  If your Python 2 scripts need additional package dependencies installed via pip, you'll need to instead use a custom Dockerfile: ARG BASE_IMAGE FROM $BASE_IMAGE RUN apt-key adv --refresh-keys --keyserver keyserver.ubuntu.com \ && apt update RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::Options::="--force-confold" --no-install-recommends --no-install-suggests python python-pip RUN pip install somepackage anotherpackage  Note: a custom Dockerfile will override webimage_extra_packages configurations in .ddev/config.yaml. Caveat 1: As of 2022, Ddev web image runs Debian 11 and sudo apt-get python still installs Python 2. This may change in future versions of Debian, so be careful when you upgrade Ddev. Caveat 2: Python 2 has reached its End Of Life and is unsupported. Additionally, important package manager pip is no longer able to natively install (without workarounds) on latest Python 2, so you're probably better off upgrading your scripts to Python 3 using the 2to3 utility. Python 3 on Ddev Use the following Ddev configuration to install Python 3 into /usr/bin/python along with most of any additional package dependencies for your py scripts. webimage_extra_packages: [python3, python-is-python3]  Note that by default, Python 3 is installed to /usr/bin/python3 so add the python-is-python3 package to make python execute Python 3. You can also usually work around needing to install the python3-pip package, because most Python 3 packages are already bundled for Debian. Therefore, additional Python 3 package dependencies can be added by comma-separated name to webimage_extra_packages. See list of stable Python packages for Debian here. If your dependencies are not bundled and you need to use pip, Conda, or another python package manager, then you must implement a custom Dockerfile at .ddev/web-build/Dockerfile like this: ARG BASE_IMAGE FROM $BASE_IMAGE RUN apt-key adv --refresh-keys --keyserver keyserver.ubuntu.com \ && apt update RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::Options::="--force-confold" --no-install-recommends --no-install-suggests python3 python-is-python3 python3-pip RUN pip3 install somepackage anotherpackage  Note: a custom Dockerfile will override webimage_extra_packages configurations in .ddev/config.yaml.
Question: I need to create a schema but it has a column called global, and when I try to write this, I got an error. class User(BaseModel): id:int global:bool  I try to use another name, but gives another error when try to save in db.

Answer: It looks like you are using a pydantic module. You can't use the name global because it's a reserved keyword so you need to use this trick to convert it. pydantic v1: class User(BaseModel): id: int global_: bool class Config: fields = { 'global_': 'global' }  or pydantic v1 & v2: class User(BaseModel): id: int global_: bool = Field(..., alias='global')  To create a class you have to use a dictionary (because User(id=1, global=False) also throws an error): user = User(id=1, global=False) > Traceback (most recent call last): > (...) > File "<input>", line 1 > User(id=1, global=False) > ^^^^^^ > SyntaxError: invalid syntax user = User(**{'id': 1, 'global': False})  Set allow_population_by_field_name = True or populate_by_name=True in config to allow creating models using both global and global_ names (thanks @GooDeeJAY). pydantic v1: class User(BaseModel): id: int global_: bool = Field(..., alias='global') class Config: allow_population_by_field_name = True  pydantic v2: class User(BaseModel): id: int global_: bool = Field(..., alias='global') model_config = ConfigDict(populate_by_name=True) user1 = User(**{'id': 1, 'global': False}) user2 = User(id=1, global_=False) assert user1 == user2  By default schema dump will not use aliased fields: user.dict() # for pydantic v1 user.model_dump() # for pydantic v2 > {'id': 1, 'global_': False}  To get data in the correct schema use by_alias: user.dict(by_alias=True) # for pydantic v1 user.model_dump(by_alias=True) # for pydantic v2 > {'id': 1, 'global': False}
Question: I`m using python poetry(https://python-poetry.org/) for dependency management in my project. Though when I`m running poetry install, its giving me below error. ERROR: Failed building wheel for numpy Failed to build numpy ERROR: Could not build wheels for numpy, which is required to install pyproject.toml-based projects  I`m having python 3.9 installed in my laptop. I installed numpy 1.21.5 using pip install numpy, I even tried to down version it to 1.19.5. Though I`m getting the same error. I found out many people are getting ERROR: Failed building wheel for numpy this error in python 3.10, they solved it by down versioning python to 3.9, though that didnt working for me.

Answer: I solved it by doing the following steps:-  I updated the pyproject.toml(This file contains all the library/dependency/dev dependency)with the numpy version that I installed using pip install numpy command.   Run poetry lock to update poetry.lock file(contains details information about the library)   Run poetry install again, & it should work fine.    In a nutshell, you just have to install the correct version of numpy Click me to check the compatibility And then install the required version using pip install numpy==version. Example: To install NumPy version 1.23.5, use the following- pip install numpy==1.23.5  If you are having any problems, you can comment. I`ll try to answer it.
Question: I have been using the ax.bar_label method to add data values to the bar graphs. The numbers are huge such as 143858918. How can I add commas to the data values using the ax.bar_label method? I do know how to add commas using the annotate method but if it is possible using bar_label, I am not sure. Is it possible using the fmt keyword argument that is available?

Answer:  Is it possible using the fmt keyword argument of ax.bar_label?  Yes, but only in matplotlib 3.7+. Prior to 3.7, fmt only accepted % formatters (no comma support), so labels was needed to f-format the container's datavalues.  If matplotlib ≥ 3.7, use fmt: for c in ax.containers: ax.bar_label(c, fmt='{:,.0f}') # ≥ 3.7 # ^no f here (not an actual f-string)   If matplotlib < 3.7, use labels: for c in ax.containers: ax.bar_label(c, labels=[f'{x:,.0f}' for x in c.datavalues]) # < 3.7     Toy example: fig, ax = plt.subplots() ax.bar(['foo', 'bar', 'baz'], [3200, 9025, 800]) # ≥ v3.7 for c in ax.containers: ax.bar_label(c, fmt='{:,.0f}')  # < v3.7 for c in ax.containers: ax.bar_label(c, labels=[f'{x:,.0f}' for x in c.datavalues])
Question: Until now the pandas function df.to_stata() worked just fine with my datasets. I am trying to export a dataframe that includes 29,778 rows and 37 to a Stata file using the following code: df.to_stata("Stata_File.dta", write_index=False, version=118)  However, I receive the following error message: AttributeError: 'DataFrame' object has no attribute 'dtype'  I would really appreciate any help how to fix this.

Answer: It's possible that this error arises when you have multiple columns with the same name in your dataframe
Question: I created a Github workflow that runs a python script with a cron schedule. On every run of the workflow an access_token is generated, which is required during the next run. To save the token the python script writes the token to the GITHUB_ENV file. In the next step, I use the hmanzur/actions-set-secret@v2.0.0 action to save the token to a Github secret. All works fine. My only problem is, that the token gets displayed in the logs of the second step as an environment variable. Here is a minimal version of the workflow file: name: Tests on: schedule: - cron: "0 1 * * *" jobs: test: runs-on: ubuntu-latest strategy: matrix: python: ['3.9'] steps: - uses: actions/checkout@v1 - uses: actions/setup-python@v1 with: python-version: ${{ matrix.python }} - name: Install dependencies run: pip install -r requirements.txt - name: Run tests working-directory: ./src run: python -m unittest env: ACCESS_TOKEN: ${{secrets.ACCESS_TOKEN}} - uses: hmanzur/actions-set-secret@v2.0.0 with: name: 'ACCESS_TOKEN' value: ${{env.ACCESS_TOKEN}} repository: Me/MyRepository token: ${{ secrets.REPO_ACCESS_TOKEN }}  I tried applying ::add-mask::. Adding echo "ACCESS_TOKEN=::add-mask::$ACCESS_TOKEN" >> $GITHUB_ENV only added ::add-mask:: to the string. Is there a way of masking all environment variables in the GITHUB_ENV file I can apply in the first step? Can I apply the masking to the variable while writing to the GITHUB_ENV file in python? Or is there a way to disable the display of the environment variables during the workflow?

Answer: Your usage of "::add-mask::" is wrong (not your fault, I hate GHA doc). What you need to do is: echo "::add-mask::$ACCESS_TOKEN" echo "ACCESS_TOKEN=$ACCESS_TOKEN" >> $GITHUB_ENV
Question: I keep on getting error message: Available subcommands: 1.0.0 Jupyter command `jupyter-nbconvert` not found.  I've tried to reinstall nbconvert using pip to no use. I've also tried the tip from this thread with installing pip install jupyter in vscode terminal but it shows that "Requirement already satisfied" VSCode fails to export jupyter notebook to html I've also tried to manually edit jupyter settings.json file to the following: "python.pythonPath": "C:\\Users\\XYZ\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\Scripts"  I've python 3.9 installed via windows store. Any tip on what might be the issue for vscode doesn't want to export the notebook?

Answer: Unsure exactly what fixed the issue but heres a summary.  Updated to python 3.10 Installed pandoc and miktex Powershell reinstall nbconvert   Received warning that nbconvert script file is installed in a location not in Path. Copied said location to System Properties - Envionment Variables - Path   Restart and install all miktex package on the go  PDF export and HTML export seems to work as intended now.
Question: I am building a simple DB interface in Python (3.9.9) and I am using psycopg (3.0.7) to connect to my Postgres (14.1) database. Until recently, the development of this app took place on Linux, but now I am using macOS Monterey on an M1 Mac mini. This seems to be causing some troubles with ctypes, which psycopg uses extensively. The error I am getting is the following: ImportError: no pq wrapper available. Attempts made: - couldn't import psycopg 'c' implementation: No module named 'psycopg_c' - couldn't import psycopg 'binary' implementation: No module named 'psycopg_binary' - couldn't import psycopg 'python' implementation: libpq library not found  Based on the source code of psycopg, this is an error of ctypes not being able to util.find_library libpq.dylib. Postgres is installed as Postgres.app, meaning that libpq.dylib's path is /Applications/Postgres.app/Contents/Versions/14/bin/lib I have tried adding this to PATH, but it did not work. I then created a symlink to the path in /usr/local/lib, but (unsurprisingly) it also did not work. I then did some digging and found this issue describing the same problem. I am not a big macOS expert, so I am unsure on how to interpret some of the points raised. Do I need to add the path to the shared cache? Also, I do not want to fork the Python repo and implement the dlopen() method as suggested, as it seems to lead to other problems. Anyhow, is there a solution to quickly bypass this problem? As an additional reference, the code producing the above error is just: import psycopg print(psycopg.__version__) 

Answer: I had this problem but the solution was suggested to me by this answer to a related question: try setting envar DYLD_LIBRARY_PATH to the path you identified. NB, to get it working myself, I:  used the path /Applications/Postgres.app/Contents/Versions/latest/lib and had to install Python 3.9
Question: I am on an apple silicon M1 MacBook Pro. I would like to have a native ARM python environment, and an environment that runs on x86 architecture with rosetta 2. I have installed two mini forge distributions, both in the home directory: miniforge3 for the native ARM installation and miniforge3_x86_64 for the x86 installation.

Answer: So far, the best solution I've found is to start the terminal with Rosetta 2, then run a function I have saved in .zshrc to initialize the correct conda installation so that I can use the correct architecture for my needs depending on the situation. My current solution is the following function named x86: x86 () { conda deactivate # >>> conda initialize >>> # !! Contents within this block are managed by 'conda init' !! __conda_setup="$('/Users/$USERNAME/miniforge3_x86_64/bin/conda' 'shell.zsh' 'hook' 2> /dev/null)" if [ $? -eq 0 ]; then eval "$__conda_setup" else if [ -f "/Users/$USERNAME/miniforge3_x86_64/etc/profile.d/conda.sh" ]; then . "/Users/$USERNAME/miniforge3_x86_64/etc/profile.d/conda.sh" else export PATH="/Users/$USERNAME/miniforge3_x86_64/bin:$PATH" fi fi unset __conda_setup # <<< conda initialize <<< export PATH="/Users/$USERNAME/miniforge3_x86_64/bin:$PATH" export PATH="/Users/$USERNAME/miniforge3_x86_64/condabin:$PATH" }  I am still feeling this out. I may add some aliases within the function as well so things like pip do not conflict, but I hope that by prepending the x86 paths the correct packages will be referenced
Question: Code: import easyocr reader = easyocr.Reader(['en']) result = reader.readtext('R.png')  Output: CUDA not available - defaulting to CPU. Note: This module is much faster with a GPU. cv2.error: Unknown C++ exception from OpenCV code  I would truly appreciate any support!

Answer: The new version of OpenCV has some issues. Uninstall the newer version of OpenCV and install the older one using: pip install opencv-python==4.5.4.60
Question: from yt_dlp import YoutubeDL with YoutubeDL() as ydl: ydl.download('https://youtu.be/0KFSuoHEYm0')  this is the relevant bit of code producing the output. what I would like to do is grab the 2nd last line from the output below, specifying the video title. I have tried a few variations of output = subprocess.getoutput(ydl)  as well as output = subprocess.Popen( ydl, stdout=subprocess.PIPE ).communicate()[0]  the output I am attempting to capture is the 2nd last line here: [youtube] 0KFSuoHEYm0: Downloading webpage [youtube] 0KFSuoHEYm0: Downloading android player API JSON [info] 0KFSuoHEYm0: Downloading 1 format(s): 22 [download] Destination: TJ Watt gets his 4th sack of the game vs. Browns [0KFSuoHEYm0].mp4 [download] 100% of 13.10MiB in 00:01  There is also documentation on yt-dlp on how to pull title from metadata or include as something in the brackets behind YoutubeDL(), but I can not quite figure it out. This is part of the first project I am making in python. I am missing an understanding of many concepts any help would be much appreciated.

Answer: Credits: answer to question: How to get information from youtube-dl in python ??  Modify your code as follows: from yt_dlp import YoutubeDL with YoutubeDL() as ydl: info_dict = ydl.extract_info('https://youtu.be/0KFSuoHEYm0', download=False) video_url = info_dict.get("url", None) video_id = info_dict.get("id", None) video_title = info_dict.get('title', None) print("Title: " + video_title) # <= Here, you got the video title  This is the output: #[youtube] 0KFSuoHEYm0: Downloading webpage #[youtube] 0KFSuoHEYm0: Downloading android player API JSON #Title: TJ Watt gets his 4th sack of the game vs. Browns
Question: I have a some python code below that walk down a tree but I want it to work down a tree checking taking some paths conditioally based on values. I want to get the LandedPrice for branches of tree based on condition and fulfillmentChannel parsed_results['LowestLanded'] = sku_multi_sku['Summary']['LowestPrices']['LowestPrice']['LandedPrice']['Amount']['value']  That walks down this tree but values because there are two LowestPrice records/dicts returned one for each condition and fulfillmentChannel pair. I want to filter on condition=new and fulfillmentChannel=Amazon so I only get back one record. When I parse XML data I can do it with code similar to LowestPrices/LowestPrice[@condition='new'][@fulfillmentChannel='Merchant']/LandedPrice/Amount" but couldn't get similar code to work here. How do I do this with dictionaries?  "LowestPrices":{ "value":"\n ", "LowestPrice":[ { "value":"\n ", "condition":{ "value":"new" #condtion new }, "fulfillmentChannel":{ "value":"Amazon" ## fulfilllmentChannel #1 }, "LandedPrice":{ "value":"\n ", "CurrencyCode":{ "value":"USD" }, "Amount":{ "value":"19.57" } }, "ListingPrice":{ "value":"\n ", "CurrencyCode":{ "value":"USD" }, "Amount":{ "value":"19.57" } }, "Shipping":{ "value":"\n ", "CurrencyCode":{ "value":"USD" }, "Amount":{ "value":"0.00" } } }, { "value":"\n ", "condition":{ "value":"new" }, "fulfillmentChannel":{ "value":"Merchant" }, "LandedPrice":{ "value":"\n ", "CurrencyCode":{ "value":"USD" }, "Amount":{ "value":"19.25" } }, "ListingPrice":{ "value":"\n ", "CurrencyCode":{ "value":"USD" }, "Amount":{ "value":"19.25" } }, "Shipping":{ "value":"\n ", "CurrencyCode":{ "value":"USD" }, "Amount":{ "value":"0.00" } } } ] }, 

Answer: You can use list comprehensions with conditional logic for your purposes like this: my_dict = { "LowestPrices": { "value": "\n ", "LowestPrice": [{ "value": "\n ", "condition": { "value": "new" }, "fulfillmentChannel": { "value": "Amazon" }, "LandedPrice": { "value": "\n ", "CurrencyCode": { "value": "USD" }, "Amount": { "value": "19.57" } }, "ListingPrice": { "value": "\n ", "CurrencyCode": { "value": "USD" }, "Amount": { "value": "19.57" } }, "Shipping": { "value": "\n ", "CurrencyCode": { "value": "USD" }, "Amount": { "value": "0.00" } } }, { "value": "\n ", "condition": { "value": "new" }, "fulfillmentChannel": { "value": "Merchant" }, "LandedPrice": { "value": "\n ", "CurrencyCode": { "value": "USD" }, "Amount": { "value": "19.25" } }, "ListingPrice": { "value": "\n ", "CurrencyCode": { "value": "USD" }, "Amount": { "value": "19.25" } }, "Shipping": { "value": "\n ", "CurrencyCode": { "value": "USD" }, "Amount": { "value": "0.00" } } } ] }, } lowest_prices = [x for x in my_dict["LowestPrices"]["LowestPrice"] if x["condition"]["value"] == "new" and x["fulfillmentChannel"]["value"] == "Amazon"]  lowest_prices is a list of all dicts that satisfy the required conditions. If you sure that you have only one dictionary in your case that satisfy conditions or you just want to get the amount of the first one, you just do this: if len(lowest_prices) > 0: amount = lowest_prices[0]["LandedPrice"]["Amount"]["value"] print(amount)
Question: I am using S3ToRedshiftOperator to load csv file into Redshift database. Kindly help to pass xcom variable to S3ToRedshiftOperator. How can we push xcom without using custom function? Error:  NameError: name 'ti' is not defined  Using below code: from airflow.operators.s3_to_redshift_operator import S3ToRedshiftOperator def export_db_fn(**kwargs): session = settings.Session() outkey = S3_KEY.format(MWAA_ENV_NAME, name[6:]) print(outkey) s3_client.put_object(Bucket=S3_BUCKET, Key=outkey, Body=f.getvalue()) ti.xcom_push(key='FILE_PATH', value=outkey) return "OK" with DAG(dag_id="export_info", schedule_interval=None, catchup=False, start_date=days_ago(1)) as dag: export_info = PythonOperator( task_id="export_info", python_callable=export_db_fn, provide_context=True ) transfer_s3_to_redshift = S3ToRedshiftOperator( s3_bucket=S3_BUCKET, s3_key="{{ti.xcom_pull(key='FILE_PATH', task_ids='export_info')}}", schema="dw_stage", table=REDSHIFT_TABLE, copy_options=['csv',"IGNOREHEADER 1"], redshift_conn_id='redshift', autocommit=True, task_id='transfer_s3_to_redshift', ) start >> export_info >> transfer_s3_to_redshift >> end 

Answer: The error message tells the problem. ti is not defined. When you set provide_context=True, Airflow makes Context available for you in the python callable. One of the attributes is ti (see source code). So you need to extract it from kwargs or set it in the function signature. Your code should be: def export_db_fn(**kwargs): ... ti = kwargs['ti'] ti.xcom_push(key='FILE_PATH', value=outkey) ...  Or if you want to use ti directly then: def export_db_fn(ti, **kwargs): ... ti.xcom_push(key='FILE_PATH', value=outkey) ...  Note: In Airflow >= 2.0 there is no need to set provide_context=True