eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: The automatic discovery of setuptools.build_meta includes top-level folders into the tarball that shouldn't be included. We were trying to build a python package with python3 -m build. Our project has a src-layout and we are using setuptools as the backend for the build. According to the documentation, the automatic discovery should figure out the project structure and build the tarball and the wheel accordingly. For the wheel build it works as expected, i.e. only the files under src/... are included on the package. On the other hand, in the tarball build also top-level folders that are on the same hierarchical level as src are included. For example, the tests and the docs folder, which shouldn't be included. Interestingly, only a few files from the docs were included, not all of them. We tried to explicitly exclude the tests and docs folder in the pyproject.toml, the setup.cfg and the MANIFEST.in, following the respective documentations, but none of them helped. We configured the build backend in the pyproject.toml like so: [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" ...  We don't have a setup.py and our setup.cfg only contains flake8 specifications. We are using setuptools-67.1.0, we tried python versions 3.8, 3.9 and 3.10. We tried it with an Ubuntu 20.04 and a Debian 11 running conda 3.8.13. The tarball and the wheel can be seen here.

Answer: Full answer: https://github.com/pypa/setuptools/issues/3883#issuecomment-1494308302 Summary:  These files are included by default in the sdist. The set of files included in a wheel is smaller. Thus, the behavior described above is expected. If you want to exclude e.g. the tests and the docs folder from the sdist, add these two lines to your MANIFEST.in:  prune tests prune docs
Question: I want to create a Telegram bot that checks for a new post on a website (currently every 15s for testing purposes). If so, it should send a message with content from the post into the Telegram channel. For this I already have the following "code skeleton": (The fine work in terms of formatting and additions comes later) import requests import asyncio from bs4 import BeautifulSoup from telegram import InputMediaPhoto from telegram.ext import Updater # Telegram Bot API Token API_TOKEN = 'XXXXXXXXXXXXXXXXX' # URL of the website URL = 'https://chemiehalle.de' # List for storing seen posts seen_posts = [] # Function for fetching posts def get_posts(): # Send request to the website res = requests.get(URL) # Parse HTML content soup = BeautifulSoup(res.content, 'html.parser') # Find all posts on the website posts = soup.find_all('article') # Iterate over each post for post in posts: # Get title of the post title = post.find('h2', class_='entry-title').text # Check if post has already been seen if title not in seen_posts: # Get image URL image_src = post.find('img')['src'] # Get short text of the post text = post.find('div', class_='entry-content clearfix').find('p').text # Send image, title, and text as message bot.bot.send_media_group(chat_id='@chemiehalleBot', media=[InputMediaPhoto(media=image_src, caption=title + '\n\n' + text)]) # Add title of the post to the list of seen posts seen_posts.append(title) # Main loop async def main(): while True: # Call get_posts function every 15s get_posts() print("Check for new posts") await asyncio.sleep(15) # Initialize Telegram Bot updater = Updater(API_TOKEN) bot = updater.bot # Start main loop asyncio.run(main())  So far I have found out that updater = Updater(API_TOKEN, use_context=True) produces errors and so I have removed use_context=True following the instructions from other posts on this site. Since that I am encountering the error TypeError: __init__() missing 1 required positional argument: 'update_queue' in the updater = Updater(API_TOKEN) line. But unfortunately I don't know what to change. According to this, the constructor of Updater needs an additional argument update_queue. But I have no idea which one this should be and where I should get it from. Can you help me please? Thank you very much for the support!

Answer: You probably have the wrong telegram version. I am a bit old fashion, but for me version 13 still works great. So simply replace your library version by running: pip install python-telegram-bot==13.13
Question: I am trying to run a Jupyter Notebook in VS Code. However, I'm getting the following error message whenever I try to execute a cell: Failed to start the Kernel. Jupyter server crashed. Unable to connect. Error code from Jupyter: 1 usage: jupyter.py [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir] [--paths] [--json] [--debug] [subcommand] Jupyter: Interactive Computing positional arguments: subcommand the subcommand to launch options: -h, --help show this help message and exit --version show the versions of core jupyter packages and exit --config-dir show Jupyter config dir --data-dir show Jupyter data dir --runtime-dir show Jupyter runtime dir --paths show all Jupyter paths. Add --json for machine-readable format. --json output paths as machine-readable json --debug output debug information about paths Available subcommands: Jupyter command `jupyter-notebook` not found. View Jupyter log for further details.  The Jupyter log referred to by the diagnostic message just contains the same text as the above diagnostic message repeated multiple times. I believe this post refers to the same issue. Unfortunately, the accepted answer does not work for me because I do not have Python: Select Interpreter to Start Jupyter server in my Command Palette. The file was working normally this morning. I also tried uninstalling and reinstalling the extensions. How can I get the Kernel to start?

Answer: This sounds like it might be a bug that found in the 2023.1 version of the Jupyter extension that affects MacOS users: Starting a Jupyter server kernel fails (when zmq does not work) #12714 (duplicates: Failed to start the Kernel, version v2023.1.2000312134 #12726, Jupyter server crashed. Unable to connect. #12746) The solution recommended there was to switch to the pre-release version while waiting for the fix to come to the regular-release-channel released the fix. Others in the threads there also found that rolling back to older extension versions also worked for them (you can rollback clicking into the down menu button next to the extension's uninstall button). If none of those solutions work, try this: pip install --upgrade --force-reinstall --no-cache-dir jupyter  And then restart VS Code. If you already have it open, you can do this in the VS Code command palette with the Developer: Reload Window command. Credit: The above command is based on this answer by @Spandana-r to the question After installing with pip, "jupyter: command not found".
Question: I was creating a Python library, I needed to compile the pyproject.toml file. I runned this command: pip-compile pyproject.toml --resolver=backtracking I got: Backend subprocess exited when trying to invoke get_requires_for_build_wheel Failed to parse .\pyproject.toml My pyproject.toml: [build-system] requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta" [project] name = "filedump" version = "1.0.0" description = "Save multiple values to a .svf file (not encrypted)" readme = "README.md" authors = [{ name = "------", email = "-------------------" }] license = { file = "LICENSE" } classifiers = [ "License :: OSI Approved :: MIT License", "Programming Language :: Python", "Programming Language :: Python :: 3", ] keywords = ["file", "encoding"] requires-python = ">=3.9" [project.optional-dependencies] dev = ["pip-tools"] [project.urls] Homepage = "https://github.com/-----------------------" [project.scripts] file = "filedump.FileOperation()"  My project root: filedump\ |-- src\ |-- |___ __init__.py |-- LICENSE |-- MANIFEST.in |-- pyproject.toml |-- README.md 

Answer: Try adding [tools.setuptools] packages = ["src"]  to your pyproject.toml See the following page for details, major thanks to Keith R. Petersen on this one :)
Question: I have a Python3 Poetry project with a pyproject.toml file specifying the dependencies: [tool.poetry.dependencies] python = "^3.10" nltk = "^3.7" numpy = "^1.23.4" scipy = "^1.9.3" scikit-learn = "^1.1.3" joblib = "^1.2.0" [tool.poetry.dev-dependencies] pytest = "^5.2"  I export those dependencies to a requirements.txt file using the command poetry export --without-hashes -f requirements.txt --output requirements.txt resulting in the following file requirements.txt: click==8.1.3 ; python_version >= "3.10" and python_version < "4.0" colorama==0.4.6 ; python_version >= "3.10" and python_version < "4.0" and platform_system == "Windows" joblib==1.2.0 ; python_version >= "3.10" and python_version < "4.0" nltk==3.8.1 ; python_version >= "3.10" and python_version < "4.0" numpy==1.24.1 ; python_version >= "3.10" and python_version < "4.0" regex==2022.10.31 ; python_version >= "3.10" and python_version < "4.0" scikit-learn==1.2.1 ; python_version >= "3.10" and python_version < "4.0" scipy==1.9.3 ; python_version >= "3.10" and python_version < "4.0" threadpoolctl==3.1.0 ; python_version >= "3.10" and python_version < "4.0" tqdm==4.64.1 ; python_version >= "3.10" and python_version < "4.0"  that I use to install the dependencies when building a Docker image. My question: How can I omit the the colorama dependency in the above list of requirements when calling poetry export --without-hashes -f requirements.txt --output requirements.txt? Possible solution: I could filter out the line with colorama by producing the requirements.txt file using poetry export --without-hashes -f requirements.txt | grep -v colorama > requirements.txt. But that seems hacky and may break things in case the Colorama requirement is expressed across multiple lines in that file. Is there a better and less hacky way? Background: When installing this list of requirements while building the Docker image using pip install -r requirements.txt I get the message Ignoring colorama: markers 'python_version >= "3.10" and python_version < "4.0" and platform_system == "Windows"' don't match your environment  A coworker thinks that message looks ugly and would like it not to be visible (but personally I don't care). A call to poetry show --tree reveals that the Colorama dependency is required by pytest and is used to make terminal colors work on Windows. Omitting the library as a requirement when installing on Linux is not likely a problem in this context.

Answer: The colorama dependency is required by pytest. I suppose your Docker image is for production use, so it doesn't have to contain pytest which is clearly a development dependency. You can use poetry export --without-hashes --without dev -f requirements.txt -o requirements.txt to prevent your dev packages to be exported to the requirements.txt file so they won't be installed by pip install. You can find some poetry export options Here.
Question: Using a completely new pip-tools setup always results in a Backend subprocess exited error. pyproject.toml: [project] dependencies = [ 'openpyxl >= 3.0.9, < 4', ]  Running pip-tools in an empty directory that only contains the above pyproject.toml: % python -m venv .venv % source .venv/bin/activate % python -m pip install pip-tools % pip-compile -v -o requirements.txt --resolver=backtracking pyproject.toml Creating venv isolated environment... Installing packages in isolated environment... (setuptools >= 40.8.0, wheel) Getting build dependencies for wheel... Backend subprocess exited when trying to invoke get_requires_for_build_wheel Failed to parse .../pyproject.toml  No requirements.txt gets created. Ideas on what might be missing here are appreciated.

Answer: Your pyproject.toml most likely is invalid, try pip install -e . and you'll see a detailed explanation. For now, pip-tools can't show a nice error message, but work is in progress.
Question: im trying to create a data set with a unique id code but i get a  'ValueError not enough values to unpack (expected 6, got 5)'  on line 8, basically, I am trying to:  generate a unique 6 digit id code append dataset value with 'ID' ex: ID123456  UPDATE: fixed the error and ID append, now how do i make sure the generated id is unique in the dataset? from faker import Faker import random import pandas as pd Faker.seed(0) random.seed(0) fake = Faker("en_US") fixed_digits = 6 concatid = 'ID' idcode,name, city, country, job, age = [[] for k in range(0,6)] for row in range(0,100): idcode.append(concatid + str(random.randrange(111111, 999999, fixed_digits))) name.append(fake.name()) city.append(fake.city()) country.append(fake.country()) job.append(fake.job()) age.append(random.randint(20,100)) d = {"ID Code":idcode, "Name":name, "Age":age, "City":city, "Country":country, "Job":job} df = pd.DataFrame(d) df.head()  planning to generate 1k rows

Answer: To answer the question:  now how do I make sure the generated id is unique in the dataset?  You have to use: unique.random_int So, your code will be like this as you see below: from faker import Faker import random import pandas as pd Faker.seed(0) random.seed(0) fake = Faker("en_US") fixed_digits = 6 concatid = 'ID' idcode,name, city, country, job, age = [[] for k in range(0,6)] for row in range(0,100): idcode.append(concatid + str(fake.unique.random_int(min=111111, max=999999))) name.append(fake.name()) city.append(fake.city()) country.append(fake.country()) job.append(fake.job()) age.append(random.randint(20,100)) d = {"ID Code":idcode, "Name":name, "Age":age, "City":city, "Country":country, "Job":job} df = pd.DataFrame(d) df.head()
Question: I'm trying to find out if Pandas.read_json performs some level of autodetection. For example, I have the following data: data_records = [ { "device": "rtr1", "dc": "London", "vendor": "Cisco", }, { "device": "rtr2", "dc": "London", "vendor": "Cisco", }, { "device": "rtr3", "dc": "London", "vendor": "Cisco", }, ] data_index = { "rtr1": {"dc": "London", "vendor": "Cisco"}, "rtr2": {"dc": "London", "vendor": "Cisco"}, "rtr3": {"dc": "London", "vendor": "Cisco"}, }  If I do the following: import pandas as pd import json pd.read_json(json.dumps(data_records)) --- device dc vendor 0 rtr1 London Cisco 1 rtr2 London Cisco 2 rtr3 London Cisco  though I get the output that I desired, the data is record based. Being that the default orient is columns, I would have not thought this would have worked. Therefore is there some level of autodetection going on? With index based inputs the behaviour seems more inline. As this shows appears to have parsed the data based on a column orient by default. pd.read_json(json.dumps(data_index)) rtr1 rtr2 rtr3 dc London London London vendor Cisco Cisco Cisco pd.read_json(json.dumps(data_index), orient="index") dc vendor rtr1 London Cisco rtr2 London Cisco rtr3 London Cisco 

Answer: TL;DR When using pd.read_json() with orient=None, the representation of the data is automatically determined through pd.DataFrame(). Explanation The pandas documentation is a bit misleading here. When not specifying orient, the parser for 'columns' is used, which is self.obj = pd.DataFrame(json.loads(json)). So pd.read_json(json.dumps(data_records))  is equivalent to pd.DataFrame(json.loads(json.dumps(data_records)))  which again is equivalent to pd.DataFrame(data_records)  I.e., you pass a list of dicts to the DataFrame constructor, which then performs the automatic determination of the data representation. Note that this does not mean that orient is auto-detected. Instead, simple heuristics (see below) on how the data should be loaded into a DataFrame are applied. Loading JSON-like data through pd.DataFrame() For the 3 most relevant cases of JSON-structured data, the DataFrame construction through pd.DataFrame() is:  Dict of lists  In[1]: data = {"a": [1, 2, 3], "b": [9, 8, 7]} ...: pd.DataFrame(data) Out[1]: a b 0 1 9 1 2 8 2 3 7   Dict of dicts  In[2]: data = {"a": {"x": 1, "y": 2, "z": 3}, "b": {"x": 9, "y": 8, "z": 7}} ...: pd.DataFrame(data) Out[2]: a b x 1 9 y 2 8 z 3 7   List of dicts  In[3]: data = [{'a': 1, 'b': 9}, {'a': 2, 'b': 8}, {'a': 3, 'b': 7}] ...: pd.DataFrame(data) Out[3]: a b 0 1 9 1 2 8 2 3 7
Question: In the context of model selection for a classification problem, while running cross validation, is it ok to specify n_jobs=-1 both in model specification and cross validation function in order to take full advantage of the power of the machine? For example, comparing sklearn RandomForestClassifier and xgboost XGBClassifier: RF_model = RandomForestClassifier( ..., n_jobs=-1) XGB_model = XGBClassifier( ..., n_jobs=-1) RF_cv = cross_validate(RF_model, ..., n_jobs=-1) XGB_cv = cross_validate(XGB_model, ..., n_jobs=-1)  is it ok to specify the parameters in both? Or should I specify it only once? And in which of them, model or cross validation statement? I used for the example models from two different libraries (sklearn and xgboost) because maybe there is a difference in how it works, also cross_validate function is from sklearn.

Answer: Specifying n_jobs twice does have an effect, though whether it has a positive or negative effect is complicated. When you specify n_jobs twice, you get two levels of parallelism. Imagine you have N cores. The cross-validation function creates N copies of your model. Each model creates N threads to run fitting and predictions. You then have N*N threads. This can blow up pretty spectacularly. I once worked on a program which needed to apply ARIMA to tens of thousands of time-series. Since each ARIMA is independent, I parallelized it and ran one ARIMA on each core of a 12-core CPU. I ran this, and it performed very poorly. I opened up htop, and was surprised to find 144 threads running. It turned out that this library, pmdarima, internally parallelized ARIMA operations. (It doesn't parallelize them well, but it does try.) I got a massive speedup just by turning off this inner layer of parallelism. Having two levels of parallelism is not necessarily better than having one. In your specific case, I benchmarked a random forest with cross validation, and I benchmarked four configurations:  No parallelism Parallelize across different CV folds, but no model parallelism Parallelize within the model, but not on CV folds Do both   (Error bars represent 95% confidence interval. All tests used RandomForestClassifier. Test was performed using cv=5, 100K samples, and 100 trees. Test system had 4 cores with SMT disabled. Scores are mean duration of 7 runs.) This graph shows that no parallelism is the slowest, CV parallelism is third fastest, and model parallelism and combined parallelism are tied for first place. However, this is closely tied to what classifiers I'm using - a benchmark for pmdarima, for example, would find that cross-val parallelism is faster than model parallelism or combined parallelism. If you don't know which one is faster, then test it.
Question: I have 2 or more HLLs that are unioned, I want to get the intersection count of that unions. I have used the example from here hll-python example Following is my code ops = [hll_ops.hll_get_union(HLL_BIN, records)] _, _, result1 = client.operate(getKey(value), ops) ops = [hll_ops.hll_get_union(HLL_BIN, records2)] _, _, result2 = client.operate(getKey(value2), ops) ops = [hll_ops.hll_get_intersect_count(HLL_BIN, [result1[HLL_BIN]] + [result2[HLL_BIN]])] _, _, resultVal = client.operate(getKey(value), ops) print(f'intersectAll={resultVal}') _, _, resultVal2 = client.operate(getKey(value2), ops) print(f'intersectAll={resultVal2}')  I get 2 different results when I use different keys for the intersection using hll_get_intersect_count, i.e resultVal and resultVal2 are not same. This does not happen in the case of union count using function hll_get_union_count. Ideally the value of intersection should be the same. Can any one tell me why is this happening and what is the right way to do it?

Answer: Was able to figure out the solutions (with the help of Aerospike support, the same question was posted here and discussed more elaboratively aerospike forum). Posting my code for others having the same issue. Intersection of HLLs is not supported in Aerospike. However, If I am to get intersection of multiple HLLs I will have to save one union into aerospike and then get intersection count of one vs the rest of the union. The key we provide in client.operate function for hll_get_intersect_count is used to get the intersection with the union. Following is the code I came up with ops = [hll_ops.hll_get_union(HLL_BIN, records)] _, _, result1 = client.operate(getKey(value), ops) # init HLL bucket ops = [hll_ops.hll_init(HLL_BIN, NUM_INDEX_BITS, NUM_MH_BITS)] _, _, _ = client.operate(getKey('dummy'), ops) # get set union and insert to inited hll bucket and save it for 5 mins(300 sec) # use hll_set_union to save the HLL into aeropike temporarily ops = [hll_ops.hll_set_union(HLL_BIN, [records2])] _, _, _ = client.operate(getKey('dummy'), ops, meta={"ttl": 300}) ops = [hll_ops.hll_get_intersect_count(HLL_BIN, [result1[HLL_BIN]])] _, _, resultVal = client.operate(getKey('dummy'), ops) print(f'intersectAll={resultVal}')  For more reference, you can look here for hll_set_union reference. More elaborate discussion can be found here
Question: Letting them compete three times (a million pops/dels each time): from timeit import timeit for _ in range(3): t1 = timeit('b.pop(0)', 'b = bytearray(1000000)') t2 = timeit('del b[0]', 'b = bytearray(1000000)') print(t1 / t2)  Time ratios (Try it online!): 274.6037053753368 219.38099365582403 252.08691226683823  Why is pop that much slower at doing the same thing?

Answer: When you run b.pop(0), Python moves all the elements back by one as you might expect. This takes O(n) time. When you del b[0], Python simply increases the start pointer of the object by 1. In both cases, PyByteArray_Resize is called to adjust the size. When the new size is smaller than half the allocated size, the allocated memory will be shrunk. In the del b[0] case, this is the only point where the data will be copied. As a result, this case will take O(1) amortized time. Relevant code: bytearray_pop_impl function: Always calls memmove(buf + index, buf + index + 1, n - index);  The bytearray_setslice_linear function is called for del b[0] with lo == 0, hi == 1, bytes_len == 0. It reaches this code (with growth == -1): if (lo == 0) { /* Shrink the buffer by advancing its logical start */ self->ob_start -= growth; /* 0 lo hi old_size | |<----avail----->|<-----tail------>| | |<-bytes_len->|<-----tail------>| 0 new_lo new_hi new_size */ } else { /* 0 lo hi old_size | |<----avail----->|<-----tomove------>| | |<-bytes_len->|<-----tomove------>| 0 lo new_hi new_size */ memmove(buf + lo + bytes_len, buf + hi, Py_SIZE(self) - hi); }