eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a requirements.txt like numpy  and an environment.yml containing # run via: conda env create --file environment.yml --- name: test dependencies: - python>=3 - pip - pip: - -r file:requirements.txt  when I then run conda env create --file environment.yml I get  Pip subprocess output: Pip subprocess error: ERROR: Exception:  <... error traceback in pip >  AttributeError: 'FileNotFoundError' object has no attribute 'read' failed CondaEnvException: Pip failed  It is also strange how pip is called, as reported just before the error occurs: ['$HOME/.conda/envs/test/bin/python', '-m', 'pip', 'install', '-U', '-r', '$HOME/test/condaenv.8d3003nm.requirements.txt']  (I replace my home path with $HOME) Note the weird expansion of the requirements.txt. Any ideas?

Answer: Changes to Pip Behavior in 21.2.1 A recent change in the Pip code has changed its behavior to be more strict with respect to file: URI syntax. As pointed out by a PyPA member and Pip developer, the syntax file:requirements.txt is not a valid URI according to the RFC8089 specification. Instead, one must either drop the file: scheme altogether: name: test dependencies: - python>=3 - pip - pip: - -r requirements.txt  or provide a valid URI, which means using an absolute path (or a local file server): name: test dependencies: - python>=3 - pip - pip: - -r file:/full/path/to/requirements.txt # - -r file:///full/path/to/requirements.txt # alternate syntax
Question: I'm trying to get a django project up and running, which depends on GDAL library. I'm working on a M1 based mac. Following the instructions on official Django docs, I've installed the necessary packages via brew $ brew install postgresql $ brew install postgis $ brew install gdal $ brew install libgeoip  gdalinfo --version runs fine and shows the version as 3.3.1 gdal-config --libs returns this path: -L/opt/homebrew/Cellar/gdal/3.3.1_2/lib -lgdal a symlink is also placed on the homebrew's lib directory, which is in my path env variable. When I try to run django without specifying the path to gdal library, it complains that it cannot find the GDAL package (even though the library is reachable, as a symlink to it is available through path env variable). When I try to specify the path to the GDAL library using GDAL_LIBRARY_PATH, I get this error: OSError: dlopen(/opt/homebrew/Cellar/gdal/3.3.1_2/lib/libgdal.dylib, 6): no suitable image found. Did find: /opt/homebrew/Cellar/gdal/3.3.1_2/lib/libgdal.dylib: mach-o, but wrong architecture /opt/homebrew/Cellar/gdal/3.3.1_2/lib/libgdal.29.dylib: mach-o, but wrong architecture  P.s. I've already seen this answer, but it didn't help. Isn't that strange when I try to run gdalinfo it runs fine but when django tries to run it throws me this error? What am I doing wrong?

Answer: GDAL and Python are likely compiled for different CPU architectures. On an M1 system the OS can run both native arm64 and emulated x86_64 binaries. To check: run file /opt/homebrew/Cellar/gdal/3.3.1_2/lib/libgdal.dylib and file $(which python3), which should show the supported CPU architectures for both. If the two don't match you'll have to reinstall one of them. Note that if you reinstall Python you also have to reinstall all Python packages with C extensions.
Question: I'm trying to open two or more separate windows. I was able to open the first window by running from selenium import webdriver import undetected_chromedriver.v2 as uc options = webdriver.ChromeOptions() options.add_argument(r"--user-data-dir=C:\Users\username\AppData\Local\Google\Chrome\User Data") drivers = list() drivers.append(uc.Chrome(options=options))  Now I tried to open the second window by simply repeating the last line (drivers.append(uc.Chrome(options=options))), but it returned RuntimeError: you cannot reuse the ChromeOptions object  So I tried options = webdriver.ChromeOptions() options.add_argument(r"--user-data-dir=C:\Users\username\AppData\Local\Google\Chrome\User Data") drivers.append(uc.Chrome(options=options))  This time it returned WebDriverException: unknown error: cannot connect to chrome at 127.0.0.1:54208 from chrome not reachable  How can I fix this?

Answer: This worked for me , I couldn't use v2 but it works in v1.  import undetected_chromedriver as uc uc.install(executable_path=PATH,) drivers_dict={} def scraping_function(link): try: thread_name= threading.current_thread().name #sometime we are going to have different thread name in each iteration so a little regex might help thread_name = re.sub("ThreadPoolExecutor-(\d*)_(\d*)", r"ThreadPoolExecutor-0_\2", thread_name) print(f"re.sub -> {thread_name}") driver = drivers_dict[thread_name] except KeyError: drivers_dict[threading.current_thread().name] = uc.Chrome(options=options,executable_path=PATH) driver = drivers_dict[threading.current_thread().name] driver.get(link)
Question: I need the pickle package installed under my Python 3.9 under Windows 10. What I tried When trying with pip install pickle I was getting:  ERROR: Could not find a version that satisfies the requirement pickle (from versions: none) ERROR: No matching distribution found for pickle  Then I tried the solution suggested in this question using pip install pickle5 but got the following error:  error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/  I then tried to install the Tool suggested by the error but got same error message after trying again pip install pickle5. Question Which is the correct way to install pickle package under Python 3.9 in Windows 10? UPDATE There is no need to install pickle module as it comes already installed along with Python 3.x. Just needed to do import pickle and voila!

Answer: Cedric UPDATED answer is right. Pickle exists within Python 3.9. You don't need pip install pickle. Just use it. import pickle
Question: I created declarative table. from sqlalchemy.ext.declarative import declarative_base from sqlalchemy import Column, String from sqlalchemy.dialects.postgresql import UUID import uuid Base = declarative_base() class User(Base): __tablename__ = 'users' id = Column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4, unique=True) name = Column(String)  I need to filter data. In the Flask-SQLAlchemy, I do name = 'foo' User.query.filter_by(name=name).first()  But if I use SQLAlchemy without Flask, then I get the error:  type object 'User' has no attribute 'query'  The only way that works for me is to filter the data through the session. engine = create_engine('DATABASE_URL') Session = sessionmaker(bind=engine) session = Session() name = 'foo' user = session.query(User).filter_by(name=name).first() session.close() 

Answer: The Model.query... idiom is not a default part of the SQLAlchemy ORM; it's a customisation provided by Flask-SQLAlchemy. It is not available in base SQLAlchemy, and that is why you get the error message.
Question: I have some python code formatters as git pre-commit hook and I have changed my python version as brew list | grep python python@3.7 python@3.9 brew unlink python@3.7 brew unlink python@3.9 brew link python@3.7 python -V Python 3.7.9  and know seems something get broken and on git commit I get env: python3.9: No such file or directory, so what is env? and how I can edit it to use python@3.7?

Answer: In .git/hooks/pre-commit I have #!/usr/bin/env python3.9  and running pre-commit install fixed it to #!/usr/bin/env python3.7
Question: I've been trying to follow an NLP notebook, and they use: from sklearn.feature_extraction import stop_words  However, this is throwing the following error: ImportError: cannot import name 'stop_words' from 'sklearn.feature_extraction'  My guess is that stop_words is not (or maybe no longer) part of the 'feature_extraction' part of sklearn, but I might be wrong. I have seen some articles that used sklearn.feature_extraction.stop_words, but at the same time I see places which have used 'text' in place of 'feature_extraction'. Am I misunderstanding something? sklearn is definitely up to date (version 0.24) and I import something from sklearn.manifold earlier in the notebook. Thanks for your help!

Answer: I have sklearn version 0.24.1, and I found that the module is now private - it's called _stop_words. So: from sklearn.feature_extraction import _stop_words  After a little digging, I found that this change was made in version 0.22, in response to this issue. It looks like they want people to use the "canonical" import for the task at hand, as described in the API docs.
Question: I am using Fedora Linux and when i want to update MongoDB tools (mongodb-org-tools) or my packages via sudo dnf update i always get error like this: Error: Problem: problem with installed package mongodb-org-database-tools-extra-4.4.4-1.el8.x86_64 - cannot install the best update candidate for package mongodb-org-database-tools-extra-4.4.4-1.el8.x86_64 - nothing provides /usr/libexec/platform-python needed by mongodb-org-database-tools-extra-4.4.5-1.el8.x86_64 - nothing provides /usr/libexec/platform-python needed by mongodb-org-database-tools-extra-4.4.6-1.el8.x86_64 - nothing provides /usr/libexec/platform-python needed by mongodb-org-database-tools-extra-4.4.7-1.el8.x86_64 (try to add '--skip-broken' to skip uninstallable packages)  i had similar error for updating mongodb and i solved them by sudo dnf upgrade mongodb-org-mongos --best --allowerasing. but still i have problem with mongodb tools

Answer: I also had problems installing Mongodb on Fedora 33. These problems occurred when I had the following code in /etc/yum.repos.d/mongodb-org.repo : [Mongodb] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/8/mongodb-org/4.4/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc  But if I use this repository instead (i.e. replace the above code in /etc/yum.repos.d/mongodb-org.repo with the code below), it all works fine: [Mongodb] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/4.4/x86_64 gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc  Next install mongodb: sudo dnf install mongodb-org  Start the service: sudo service start mongod  After starting the service as above, you can then use the usual systemctl commands to stop, start and show status of the service. The above command to start the service is needed only once. sudo systemctl stop mongod sudo systemctl start mongod sudo systemctl status mongod  Further note on Fedora 34: The above does not work on Fedora 34 as mongodb-org-shell dependencies on an older version of openssl causes problems: - nothing provides libcrypto.so.10()(64bit) needed by mongodb-org-shell-4.4.0-1.amzn1.x86_64 - nothing provides libssl.so.10()(64bit) needed by mongodb-org-shell-4.4.0-1.amzn1.x86_64  I conclude from https://jira.mongodb.org/browse/SERVER-58870 that the Mongodb team do not plan to support their product on Fedora going forward, as Mongodb 5.0 is also not supported on Fedora 34, although a workaround is suggested. I'm therefore going to consider other NoSQL options.
Question: System : Windows 10 Python : 3.9.5 I was learning to Deploying a Flask app on Google Cloud. I was trying to install uwsgi (on my windows system) as shown in this youtube video. pip install uwsgi  This error is coming  (flask) D:\projects\websites\googleHostFlaskApp>pip install uwsgi Collecting uwsgi Using cached uWSGI-2.0.19.1.tar.gz (803 kB) ERROR: Command errored out with exit status 1: command: 'd:\projects\websites\googlehostflaskapp\env\scripts\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\HP\\AppData\\Local\\Temp\\pip-install-93528com\\uwsgi_9bf2ba795b9549d6a9a0d2dfa78cd774\\setup.py'"'"'; __file__='"'"'C:\\Users\\HP\\AppData\\Local\\Temp\\pip-install-93528com\\uwsgi_9bf2ba795b9549d6a9a0d2dfa78cd774\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\HP\AppData\Local\Temp\pip-pip-egg-info-tiy5mnl9' cwd: C:\Users\HP\AppData\Local\Temp\pip-install-93528com\uwsgi_9bf2ba795b9549d6a9a0d2dfa78cd774\ Complete output (7 lines): Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Users\HP\AppData\Local\Temp\pip-install-93528com\uwsgi_9bf2ba795b9549d6a9a0d2dfa78cd774\setup.py", line 3, in <module> import uwsgiconfig as uc File "C:\Users\HP\AppData\Local\Temp\pip-install-93528com\uwsgi_9bf2ba795b9549d6a9a0d2dfa78cd774\uwsgiconfig.py", line 8, in <module> uwsgi_os = os.uname()[0] pi ---------------------------------------- WARNING: Discarding https://files.pythonhosted.org/packages/c7/75/45234f7b441c59b1eefd31ba3d1041a7e3c89602af24488e2a22e11e7259/uWSGI-2.0.19.1.tar.gz#sha256=faa85e053c0b1be4d5585b0858d3a511d2cd10201802e8676060fd0a109e5869 (from https://pypi.org/simple/uwsgi/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.  Answers i tried : For windows : I am not sure where we have to place the file given in the answer ? For ubuntu : I get that i have to add some development library to python, but not able to find the correct one here to install it in my windows system.

Answer: Installing UWSGI wasn't an easy task. @Seraph answer did helped but he missed some points, that i did to finally install UWSGI. So here is Complete Steps so that you won't have to waste a day on this like me:  Install Cygwin download (Cygwin is an open source collection of tools that allows Unix or Linux applications to be compiled and run on a Windows operating system from within a Linux-like interface). This youtube link will help you in the process of downloading Cygwin.  Packages u require in cygwin : gcc-core gcc-g++ libintl-devel python3-devel python38-devel gettext-devel   (if it gives error that it require c compiler, then gcc-core will help, if it errors: -lintl missing, then libintl-devel helps) 3.After you open the Cygwin app, you can follow the above youtube link, to successfully run uwsgi on your Windows System.
Question: I want to print formatted numpy array along with a float with different significant figures. Consider the following code a = 3.14159 X = np.array([1.123, 4.456, 7.789]) print('a = %4.3f, X = %3.2f' % (a, X)) ------------------------ TypeError: only size-1 arrays can be converted to Python scalars  I desire following output- a = 3.141, X = [1.12, 4.45, 7.78]  Suggest a modification in the code.

Answer: Convert array to string first with array2string: print('a = %4.3f, X = %s' % (a, np.array2string(X, precision=2))) # a = 3.142, X = [1.12 4.46 7.79]
Question: I am currently working on a python application, that works with facebook api's. As we all know, facebook loves their own technology and is working with zstd for data compression. The problem: facebook is returning either a uncompressed response with normal json or if the response is longer, it is responding with a zstd compressed json. My current code is something like this: import zstd import json def handle_response(response) json = None try: json = json.loads(zstd.decompress(response.content)) except: json = json.loads(response.text) return json  I am currently wondering, if there is a more clean way to do this and even detect zstd.

Answer: What you're doing is fine. You could, I suppose, check to see if the stream starts with the four bytes 28 b5 2f fd. If it doesn't, it's not a zstd stream. If it does, it may be a zstd stream. In the latter case, you would try to decompress and if it fails, you would fall back to just copying the input. That turns out to be exactly the same as what you're already doing, since the first thing that zstd.decompress is doing is to look for that signature.
Question: My code is not working in vscode when I click to run code I see this error:  Sorry, something went wrong activating IntelliCode support for Python. Please check the "Python" and "VS IntelliCode" output windows for details.  and when I tried to rerun the code I saw this message;  Code is already running  The code doesn't stop when I click on ctrl+c so I have to close the editor and open it back. I don't understand why this happened, please help.

Answer: I would just like to add a few helpful links: Intellicode Issue 57 Intellicode Issue 266 Gitmemory issue 486082039 For a lot of people, it just began working after a few tries randomly. See this text (quoted from issue 57):  There's a race condition in the activation of both the IntelliCode and Python language server extensions. Even if the Python extension is loaded, the language server that the extension spins up might not be fully initialized yet. So if the Python extension loads, then the IntelliCode extension, then the Python language server initializes, we will have this problem.  For some people, it was working to reload VS Intellicode pack following the reinstall the Python extension pack. Thank you.
Question: I am trying to implement my own str.join method in Python, e.g: ''.join(['aa','bbb','cccc']) returns 'aabbbcccc'. I know that string concatenation using the join method would result in linear (in the number of characters of the result) complexity, and I want to know how to do it, as using the '+' operator in a for loop would result in quadratic complexity e.g.: res='' for word in ['aa','bbb','cccc']: res = res + word  As strings are immutable, this copies a new string at each iteration resulting in quadratic running time. However, I want to know how to do it in linear time or find how ''.join works exactly. I could not find anywhere a linear time algorithm nor the implementation of str.join(iterable). Any help is much appreciated.

Answer: Joining str as actual str is a red herring and not what Python itself does: Python operates on mutable bytes, not the str, which also removes the need to know string internals. In specific, str.join converts its arguments to bytes, then pre-allocates and mutates its result. This directly corresponds to:  a wrapper to encode/decode str arguments to/from bytes summing the len of elements and separators allocating a mutable bytesarray to construct the result copying each element/separator directly into the result  # helper to convert to/from joinable bytes def str_join(sep: "str", elements: "list[str]") -> "str": joined_bytes = bytes_join( sep.encode(), [elem.encode() for elem in elements], ) return joined_bytes.decode() # actual joining at bytes level def bytes_join(sep: "bytes", elements: "list[bytes]") -> "bytes": # create a mutable buffer that is long enough to hold the result total_length = sum(len(elem) for elem in elements) total_length += (len(elements) - 1) * len(sep) result = bytearray(total_length) # copy all characters from the inputs to the result insert_idx = 0 for elem in elements: result[insert_idx:insert_idx+len(elem)] = elem insert_idx += len(elem) if insert_idx < total_length: result[insert_idx:insert_idx+len(sep)] = sep insert_idx += len(sep) return bytes(result) print(str_join(" ", ["Hello", "World!"]))  Notably, while the element iteration and element copying basically are two nested loops, they iterate over separate things. The algorithm still touches each character/byte only thrice/once.
Question: I have a list, and I need to find the maximum element in the list and also record the index at which that max element is at. This is my code. list_c = [-14, 7, -9, 2] max_val = 0 idx_max = 0 for i in range(len(list_c)): if list_c[i] > max_val: max_val = list_c[i] idx_max = list_c.index(i) return list_c, max_val, idx_max  Please help. I am just beginning to code and got stuck here.

Answer:  You are trying to find the index of i. It should be list_c[i].   Easy way/Better way is: idx_max = i. (Based on @Matthias comment above.)   Use print to print the results and not return. You use return inside functions. Also your code doesn't work if list_c has all negative values because you are setting max_val to 0. You get a 0 always.  list_c = [-14, 7, -9, 2] max_val = 0 idx_max = 0 for i in range(len(list_c)): if list_c[i] > max_val: max_val = list_c[i] idx_max = i print(list_c, max_val, idx_max)  [-14, 7, -9, 2] 7 1
Question: I have a DAG that takes the results of a script in a Google cloud bucket, loads it into a table in Google BigQuery, then deletes the file in the bucket. I want the DAG to check every hour over the weekends. Right now, I'm using a GoogleCloudStoragetoBigQueryOperator. If the file is not there, the DAG fails. Is there a way I can set up the DAG to where it won't fail if the file isn't there? Maybe a try/catch?

Answer: You could use GCSObjectExistenceSensor from Google provider package in order to verify if the file is present before running downstream tasks. gcs_object_exists = GCSObjectExistenceSensor( bucket=BUCKET_1, object=PATH_TO_UPLOAD_FILE, mode='poke', task_id="gcs_object_exists_task", )  You can check the official example here. Keep in mind this sensor extends from BaseSensorOperator so you can define params such as poke_interval, timeout and mode to suit your needs.   soft_fail (bool) – Set to true to mark the task as SKIPPED on failure poke_interval (float) – Time in seconds that the job should wait in between each tries timeout (float) – Time, in seconds before the task times out and fails. mode (str) – How the sensor operates. Options are: { poke | reschedule }, default is poke. When set to poke the sensor is taking up a worker slot for its whole execution time and sleeps between pokes. Use this mode if the expected runtime of the sensor is short or if a short poke interval is required. Note that the sensor will hold onto a worker slot and a pool slot for the duration of the sensor’s runtime in this mode. When set to reschedule the sensor task frees the worker slot when the criteria is not yet met and it’s rescheduled at a later time. Use this mode if the time before the criteria is met is expected to be quite long. The poke interval should be more than one minute to prevent too much load on the scheduler. exponential_backoff (bool) – allow progressive longer waits between pokes by using exponential backoff algorithm   source
Question: My application requires a starting matrix where each column is staggered-by-1 from the previous. It will contain millions of complex numbers representing a signal, but a small example is: array([[ 0, 1, 2, 3], [ 1, 2, 3, 4], [ 2, 3, 4, 5], [ 3, 4, 5, 6], [ 4, 5, 6, 7], [ 5, 6, 7, 8], [ 6, 7, 8, 9], [ 7, 8, 9, 10]])  I tried two creation methods, one fast, one slow. I don't understand why the fast matrix creation method causes subsequent calculations to run slowly, while the slow matrix creation results in faster running calculations. The subroutine calcs() simply takes FFTs to offer minimal code to demonstrate the issue I see in my actual signal processing code. A sample run yields: python ex.py Slow Create, Fast Math 57.90 ms, create 36.79 ms, calcs() 94.69 ms, total Fast Create, Slow Math 15.13 ms, create 355.38 ms, calcs() 370.50 ms, total  Code follows. Any insight would be appreciated! import numpy as np import time N = 65536 Np = 64 # Random signal for demo. x = np.random.randint(-50,50,N+Np) + 1j*np.random.randint(-50,50,N+Np) def calcs(sig): np.fft.fft(sig) print('Slow Create, Fast Math') t0 = time.time() X = np.zeros((N, Np), dtype=complex) for col in range(Np): X[:,col] = x[col:col+N] t1 = time.time() calcs(X) t2 = time.time() print(' %6.2f ms, create' % (1e3 * (t1 - t0))) print(' %6.2f ms, calcs()' % (1e3 * (t2 - t1))) print(' %6.2f ms, total' % (1e3 * (t2 - t0))) print('Fast Create, Slow Math') t0 = time.time() X = np.array([x[i:i+N] for i in range(Np)]).transpose() t1 = time.time() calcs(X) t2 = time.time() print(' %6.2f ms, create' % (1e3 * (t1 - t0))) print(' %6.2f ms, calcs()' % (1e3 * (t2 - t1))) print(' %6.2f ms, total' % (1e3 * (t2 - t0))) 

Answer: user3483203's comment, above, provides answer to the issue. If I avoid the transpose by creating the matrix with: X = np.array([x[i:i+Np] for i in range(N)], dtype=complex)  subsequent calcs() timing is as expected. Thank you, user3483203!