eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I use pip-tools to manage my dependencies and environments which perfectly generates a requirements.txt file for my package that consists of a setup.py that looks like this: #! /usr/bin/env python import os from setuptools import setup if "CI_COMMIT_TAG" in os.environ: VERSION = os.environ["CI_COMMIT_TAG"] else: VERSION = "0.0.0" setup(version=VERSION)  and a setup.cfg like this: ... [options] python_requires = >=3.7 zip_safe = False packages = find: include_package_data = True install_requires = PyYAML Cython numpy==1.17.5 pandas==0.25.3 ... package_dir= foo=bar [options.extras_require] testing = tox>=3.1.2 pytest>=3.4.0 coverage = coverage pytest-cov>=2.5.1 other = anybadge ...  Running $ pip-compile --index-url https://foo@bar@gitlab.my.host/api/v4/projects/236/packages/pypi/simple --no-header --allow-unsafe  yields my package requirements: ... async-timeout==3.0.1 # via aiohttp attrs==21.2.0 # via aiohttp bcrypt==3.2.0 ...  But this only includes all the packages from the install_requires section of my setup.cfg file and not the requirements from extras_require. It should work with a dev_requirements.in file as described here but I would rather only use one configuration file. How can I create a separate dev_requirements.txt from this extras_require section of my setup.cfg file using pip-compile without having to create a dev_requirements.in file? Thanks in advance!

Answer: After digging for a while, I found my answer in another issue: $ pip-compile --extra testing --extra other
Question: Yesterday I could run and debug my Azure Function project with VS Code. Today, when I'm hitting F5 ("Start Debugging"), a pop-up instantly appears with  connect ECONNREFUSED 127.0.0.1:9091  I believe it's a network issue, since VS Code doesn't even open a terminal displaying "Executing task [...]". But within VS Code I can browse the extension marketplace, run "pip install" within the terminal. Here is my configuration: launch.json { "version": "0.2.0", "configurations": [ { "name": "Python: Current file", "type": "python", "request": "launch", "program": "${file}", "console": "integratedTerminal" }, { "name": "Azure Functions", "type": "python", "request": "attach", "port": 9091, "preLaunchTask": "func: host start" } ] }  tasks.json { "version": "2.0.0", "tasks": [ { "type": "func", "command": "host start", "problemMatcher": "$func-python-watch", "isBackground": true, "dependsOn": "pipInstall", "options": { "cwd": "${config:azureFunctions.projectSubpath}" } }, { "label": "pipInstall", "type": "shell", "command": "${config:python.pythonPath} -m pip install -r requirements.txt", "problemMatcher": [], "options": { "cwd": "${config:azureFunctions.projectSubpath}" } } ] }  settings.json { "azureFunctions.deploySubpath": "azure-functions\\my_project", "azureFunctions.projectSubpath": "azure-functions\\my_project", "azureFunctions.scmDoBuildDuringDeployment": true, "azureFunctions.pythonVenv": ".venv", "azureFunctions.projectLanguage": "Python", "azureFunctions.projectRuntime": "~3", "debug.internalConsoleOptions": "neverOpen", "python.pythonPath": "C:\\Users\\myself\\BigRepo\\azure-functions\\my_project\\.venv\\Scripts\\python.exe", "powershell.codeFormatting.addWhitespaceAroundPipe": true //"terminal.integrated.shell.windows": "&&" } 

Answer: This is so weird. I edited my launch.json back and forth, sometimes it was a valid one, sometimes it was not, and now when I'm saving my original launch.json, the configuration works. This configuration works fine: { "version": "0.2.0", "configurations": [ { "name": "Python: Current file", "type": "python", "request": "launch", "program": "${file}", "console": "integratedTerminal" }, { "name": "Azure Functions", "type": "python", "request": "attach", "port": 9091, "preLaunchTask": "func: host start" } ] }  I'm not sure I understand what is going on.
Question: I'm currently studying CS231 assignments and I've realized something confusing. When calculating gradients, when I first reshape x then get transpose I got the correct result. x_r=x.reshape(x.shape[0],-1) dw= x_r.T.dot(dout)   However, when I reshape directly as the X.T shape it doesn't return the correct result. dw = x.reshape(-1,x.shape[0]).dot(dout)   Can someone explain the following question? How does the order of getting elements with np.reshape() change? How reshaping (N,d1,d2..dn) shaped array into N,D array differs from getting a reshaped array of (D,N) with its transpose.

Answer: While both your approaches result in arrays of same shape, there will by a difference in the order of elements due to the way numpy reads / writes elements. By default, reshape uses a C-like index order, which means the elements are read / written with the last axis index changing fastest, back to the first axis index changing slowest (taken from the documentation). Here is an example of what that means in practice. Let's assume the following array x: x = np.asarray([[[1, 2], [3, 4], [5, 6]], [[7, 8], [9, 10], [11, 12]]]) print(x.shape) # (2, 3, 2) print(x) # output [[[ 1 2] [ 3 4] [ 5 6]] [[ 7 8] [ 9 10] [11 12]]]  Now let's reshape this array the following two ways: opt1 = x.reshape(x.shape[0], -1) opt2 = x.reshape(-1, x.shape[0]) print(opt1.shape) # outptu: (2, 6) print(opt2.shape) # output: (6, 2) print(opt1) # output: [[ 1 2 3 4 5 6] [ 7 8 9 10 11 12]] print(opt2) # output: [[ 1 2] [ 3 4] [ 5 6] [ 7 8] [ 9 10] [11 12]]  reshape first inferred the shape of the new arrays and then returned a view where it read the elements in C-like index order. To exemplify this on opt1: since the original array x has 12 elements, it inferred that the new array opt1 must have a shape of (2, 6) (because 2*6=12). Now, reshape returns a view where: opt1[0][0] == x[0][0][0] opt1[0][1] == x[0][0][1] opt1[0][2] == x[0][1][0] opt1[0][3] == x[0][1][1] opt1[0][4] == x[0][2][0] opt1[0][5] == x[0][2][1] opt1[1][0] == x[1][0][0] ... opt1[1][5] == x[1][2][1]  So as described above, the last axis index changes fastest and the first axis index slowest. In the same way, the output for opt2 will be computed. You can now verify that transposing the first option will result in the same shape but a different order of elements: opt1 = opt1.T print(opt1.shape) # output: (6, 2) print(opt1) # output: [[ 1 7] [ 2 8] [ 3 9] [ 4 10] [ 5 11] [ 6 12]]  Obviously, the two approaches do not result in the same array due to element ordering, even though they will have the same shape.
Question: I am writing a python post request with a bytes body: with open('srt_file.srt', 'rb') as f: data = f.read() res = requests.post(url='http://localhost:8000/api/parse/srt', data=data, headers={'Content-Type': 'application/octet-stream'})  And in the server part, I tried to parse the body: app = FastAPI() BaseConfig.arbitrary_types_allowed = True class Data(BaseModel): data: bytes @app.post("/api/parse/{format}", response_model=CaptionJson) async def parse_input(format: str, data: Data) -> CaptionJson: ...  However, I got the 422 error: {"detail":[{"loc":["body"],"msg":"value is not a valid dict","type":"type_error.dict"}]} So where is wrong with my code, and how should I fix it? Thank you all in advance for helping out!!

Answer: FastAPI by default will expect you to pass json which will parse into a dict. It can't do that if it's isn't json, which is why you get the error you see. You can use the Request object instead to receive arbitrary bytes from the POST body. from fastapi import FastAPI, Request app = FastAPI() @app.get("/foo") async def parse_input(request: Request): data: bytes = await request.body() # Do something with data  You might consider using Depends which will allow you to clean up your route function. FastAPI will first call your dependency function (parse_body in this example) and will inject that as an argument into your route function. from fastapi import FastAPI, Request, Depends app = FastAPI() async def parse_body(request: Request): data: bytes = await request.body() return data @app.get("/foo") async def parse_input(data: bytes = Depends(parse_body)): # Do something with data pass
Question: I'm currently have an issue with importing the library pandas to my AWS Lambda Function. I have tried two scenarios.  Installing pandas directly into one folder with my lambda_function and uploading the zipped file.  Creating a layer with an uploaded zip file with the following structure:   - python - lib - python3.8 - site-packages - all the pandas packages here  My lambda_function is just: import json import pandas as pd def lambda_handler(event, context): return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') }  This is my error: START RequestId: 9e27641e-587b-4be2-b9be-c9be85007f9e Version: $LATEST [ERROR] Runtime.ImportModuleError: Unable to import module 'main': Unable to import required dependencies: numpy: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE! Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed. We have compiled some common reasons and troubleshooting tips at: https://numpy.org/devdocs/user/troubleshooting-importerror.html Please note and check the following: * The Python version is: Python3.8 from "/var/lang/bin/python3.8" * The NumPy version is: "1.21.1" and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help. Original error was: No module named 'numpy.core._multiarray_umath'  Is there any other approach? I don't want to use Docker for this task. Thanks!

Answer: I have solved the issue, thanks to this article: https://korniichuk.medium.com/lambda-with-pandas-fd81aa2ff25e In my case, I cannot normally install the libraries through pip, I'm on a windows machine. You must install the linux versions of pandas and numpy. Since I'm on python 3.8 I installed these versions:  numpy-1.21.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl pandas-1.3.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl  After downloading the packages, I replaced the pandas and numpy folders that were originally from the install via pip install pandas. I used my first scenario as showed in my question.
Question: I have: foo/ ├── __init__.py ├── bar.py └── baz ├── __init__.py └── alice.py  In bar.py, I import Alice, which is an empty class with nothing in it but the name attribute set to "Alice". from baz.alice import Alice a = Alice() print(a.name)  This runs properly: $ python foo/bar.py Alice  But mypy complains: $ mypy --version mypy 0.910 $ mypy --strict . foo/bar.py:1: error: Cannot find implementation or library stub for module named "baz.alice" foo/bar.py:1: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 6 source files)  Why is mypy complaining?

Answer: mypy has its own search path for imports and does not resolve imports exactly as Python does and it isn't able to find the baz.alice module. Check the documentation listed in the error message, specifically the section on How imports are found:  The rules for searching for a module foo are as follows: The search looks in each of the directories in the search path (see above) until a match is found.  If a package named foo is found (i.e. a directory foo containing an __init__.py or __init__.pyi file) that’s a match. If a stub file named foo.pyi is found, that’s a match. If a Python module named foo.py is found, that’s a match.   The documentation also states that this in the section on Mapping file paths to modules:  For each file to be checked, mypy will attempt to associate the file (e.g. project/foo/bar/baz.py) with a fully qualified module name (e.g. foo.bar.baz).  There's a few ways to solve this particular issue:  As paul41 mentioned in his comment, one option to solve this issue is by providing the fully qualified import (from foo.baz.alice import Alice), and then running from a top-level module (a .py file in the root level). You could add a # type: ignore to the import line. You can edit the MYPYPATH variable to point to the foo directory:  (venv) (base) ➜ mypy foo/bar.py --strict foo/bar.py:3: error: Cannot find implementation or library stub for module named "baz.alice" foo/bar.py:3: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports Found 1 error in 1 file (checked 1 source file) (venv) (base) ➜ export MYPYPATH=foo/ (venv) (base) ➜ mypy foo/bar.py --strict Success: no issues found in 1 source file
Question: The Python Packaging Tutorial recommends that "Static metadata (setup.cfg) should be preferred. Dynamic metadata (setup.py) should be used only as an escape hatch when absolutely necessary. setup.py used to be required, but can be omitted with newer versions of setuptools and pip." The guide to packaging and distributing projects explains that "setup.cfg is an ini file that contains option defaults for setup.py commands. For an example, see the setup.cfg in the PyPA sample project." That example is entirely useless, and there doesn't appear to be a lot of other helpful information. The examples in the tutorial suggest that some, or perhaps all, valid arguments to setuptools.setup() can be listed in setup.cfg, but there is no real explanation to this effect. In particular, it's not clear how to translate a list argument, like that for the very common and important install_requires parameter, into lines in setup.cfg. The proper way to do this, as I've determined by deduction and experimentation, appears to be as follows: [options] install_requires = dependency_1 dependency_2  Obviously it would be better for this to be properly documented somewhere so that new package creators do not have to undergo a similar process just to specify their projects' dependencies. Does such documentation exist?

Answer: Yes, in the documentation of the setuptools. Here it is: https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html
Question: I am using Word2Vec and using a wiki trained model that gives out the most similar words. I ran this before and it worked but now it gives me this error even after rerunning the whole program. I tried to take off return_path=True but im still getting the same error print(api.load('glove-wiki-gigaword-50', return_path=True)) model.most_similar("glass")  #ERROR: /Users/me/gensim-data/glove-wiki-gigaword-50/glove-wiki-gigaword-50.gz --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-153-3bf32168d154> in <module> 1 print(api.load('glove-wiki-gigaword-50', return_path=True)) ----> 2 model.most_similar("glass") AttributeError: 'Word2Vec' object has no attribute 'most_similar'  #MODEL this is the model I used  print( '%s (%d records): %s' % ( model_name, model_data.get('num_records', -1), model_data['description'][:40] + '...', ) )  Edit: here is my gensim download & output !python -m pip install -U gensim  OUTPUT: Requirement already satisfied: gensim in ./opt/anaconda3/lib/python3.8/site-packages (4.0.1) Requirement already satisfied: numpy>=1.11.3 in ./opt/anaconda3/lib/python3.8/site-packages (from gensim) (1.20.1) Requirement already satisfied: smart-open>=1.8.1 in ./opt/anaconda3/lib/python3.8/site-packages (from gensim) (5.1.0) Requirement already satisfied: scipy>=0.18.1 in ./opt/anaconda3/lib/python3.8/site-packages (from gensim) (1.6.2)

Answer: You are probably looking for <MODEL>.wv.most_similar, so please try: model.wv.most_similar("glass")
Question: I'm trying to use the new bar_label option in Matplotlib but am unable to find a way to append text e.g. '%' after the label values. Previously, using ax.text I could use f-strings, but I can't find a way to use f-strings with the bar-label approach. fig, ax = plt.subplots(1, 1, figsize=(12,8)) hbars = ax.barh(wash_needs.index, wash_needs.values, color='#2a87c8') ax.tick_params(axis='x', rotation=0) # previously I used this approach to add labels #for i, v in enumerate(wash_needs): # ax.text(v +3, i, str(f"{v/temp:.0%}"), color='black', ha='right', va='center') ax.bar_label(hbars, fmt='%.2f', padding=3) # this adds a label but I can't find a way to append a '%' after the number plt.show() 

Answer: I found a way to append '%' to the label figures - add an additional '%%' ax.bar_label(hbars, fmt='%.2f%%', padding=3)  Working Example import pandas as pd import seaborn as sns # for tips data tips = sns.load_dataset('tips').loc[:15, ['total_bill', 'tip']] tips.insert(2, 'tip_percent', tips.tip.div(tips.total_bill).mul(100).round(2)) total_bill tip tip_percent 0 16.99 1.01 5.94 1 10.34 1.66 16.05 2 21.01 3.50 16.66 3 23.68 3.31 13.98 4 24.59 3.61 14.68 # plot ax = tips.plot(kind='barh', y='tip_percent', legend=False, figsize=(12, 8)) labels = ax.set(xlabel='Tips: Percent of Bill (%)', ylabel='Record', title='Tips % Demo') annotations = ax.bar_label(ax.containers[0], fmt='%.2f%%', padding=3) ax.margins(x=0.1)
Question: I am using postgres with docker and having trouble with it. I successfully docker-compose up --build When I run below command it works fine. psql starts with my_username user as expected. I can see my database, \l, \dt commands works ok.  docker-compose exec db psql --username=my_username --dbname=my_database  But when I run below commands I get error role “postgres” does not exist, Additionally \l, \dt not works, even psql command not works  docker exec -it my_db_container_1 bash su - postgres createuser --createdb --password new_user  How can I get it right in the second case? What is going wrong? I am confused docker-compose.yml version: "3.9" services: #### other services #### db: image: postgres:latest restart: always environment: POSTGRES_DB: my_database POSTGRES_USER: my_username POSTGRES_PASSWORD: my_password ports: - 5432 volumes: - postgres_data:/var/lib/postgresql/data/ volumes: postgres_data: 

Answer: You have changed the default username/database/password that the postgres database is initialized with by providing the POSTGRES_USER, POSTGRES_DB, and POSTGRES_PASSWORD environment variables. When you run createuser without a -U option, it tries to connect as the current user (postgres in this case) which doesn't exist on the database because you initialized it with the user my_username. The reason docker-compose exec db psql --username=my_username --dbname=my_database works is because you are correctly supplying the username and database names that the database was initialized with. If you remove the POSTGRES_USER and POSTGRES_DB environment variables, it will initialize with the defaults postgres/postgres. Note that because you are mounting a volume into this container, which will now have an initialized database in it already, it will not be reinitialized even if you restart your compose. You need to docker volume rm that volume in order to have the database be reinitialized when you start the container.
Question: I am trying to parse JSON input as string in Python, not able to parse as list or dict since the JSON input is not in a proper format (Due to limitations in the middleware can't do much here.) { "Records": "{Output=[{_fields=[{Entity=ABC , No=12345, LineNo= 1, EffDate=20200630}, {Entity=ABC , No=567, LineNo= 1, EffDate=20200630}]}" }  I tried json.loads and ast.literal (invalid syntax error). How can I load this?

Answer: If the producer of the data is consistent, you can start with something like the following, that aims to bridge the JSON gap. import re import json source = { "Records": "{Output=[{_fields=[{Entity=ABC , No=12345, LineNo= 1, EffDate=20200630}, {Entity=ABC , No=567, LineNo= 1, EffDate=20200630}]}" } s = source["Records"] # We'll start by removing any extraneous white spaces s2 = re.sub('\s', '', s) # Surrounding any word with " s3 = re.sub('(\w+)', '"\g<1>"', s2) # Replacing = with : s4 = re.sub('=', ':', s3) # Lastly, fixing missing closing ], } ## Note that }} is an escaped } for f-string. s5 = f"{s4}]}}" >>> json.loads(s5) {'Output': [{'_fields': [{'Entity': 'ABC', 'No': '12345', 'LineNo': '1', 'EffDate': '20200630'}, {'Entity': 'ABC', 'No': '567', 'LineNo': '1', 'EffDate': '20200630'}]}]}  Follow up with some robust testing and have a nice polished ETL with your favorite tooling.
Question: The code is as below. driver = webdriver.Chrome(chromedriver_path) #webdriver path driver.get('https://webtoon.kakao.com/content/%EB%B0%94%EB%8B%88%EC%99%80-%EC%98%A4%EB%B9%A0%EB%93%A4/1781') #website access time.sleep(2) driver.execute_script("window.scrollTo(0, 900)") #scroll down time.sleep(1)  However, the page does not scroll. How can I scroll? Page link to be scrolled

Answer: Tried with the below code, it did scroll. driver.get("https://webtoon.kakao.com/content/%EB%B0%94%EB%8B%88%EC%99%80-%EC%98%A4%EB%B9%A0%EB%93%A4/1781") time.sleep(2) options = driver.find_element_by_xpath("//div[@id='root']/main/div/div/div/div[1]/div[3]/div/div/div[1]/div/div[2]/div/div[1]/div/div/div/div") driver.execute_script("arguments[0].scrollIntoView(true);",options)
Question: I'm trying to keep a singular color palette in my thesis, and I noticed that the blue of my histplots and the blue of my countplots are slightly different shades, even though I set them to the exact same hexadecimal value. Is there a setting that I'm missing or do these different plots not just show the hexadecimal as given? I've tried playing around with the countplot saturation but it doesn't match the color. Ideally all of my histplots would have the same color as my countplots (and bar plots which use the countplot coloring too). Below a minimum code example: import seaborn as sns import matplotlib.pyplot as plt sns.set(rc={'figure.figsize':(20,10)}, font_scale=2) plt.rcParams['axes.grid'] = False titanic = sns.load_dataset('titanic') fig, ax = plt.subplots(1,2) sns.countplot(x="class", data=titanic, ax=ax[0], color='#5975a4') sns.histplot(x="who", data=titanic, ax=ax[1], color='#5975a4')  It produces the following figure: 

Answer: The countplot has a saturation parameter (more saturation is more "real" color, less saturation is closer to grey). Seaborn uses saturation in bar plots to make the default colors look "smoother". The default saturation is 0.75; it can be set to 1 to get the "true" color. The histplot has an alpha parameter, making the color semi-transparent. The color gets mixed with the background, so it looks different depending on the background color. In this case, the alpha seems to default to 0.75. As that also has an effect similar to saturation, the histplot doesn't use saturation. The transparency is especially useful when multiple histograms are drawn in the same subplot. To get both in "real" color, set both the saturation of the countplot and the alpha of the histplot to 1: import seaborn as sns import matplotlib.pyplot as plt sns.set(rc={'figure.figsize': (20, 10)}, font_scale=2) plt.rcParams['axes.grid'] = False titanic = sns.load_dataset('titanic') fig, ax = plt.subplots(1, 2) sns.countplot(x="class", data=titanic, ax=ax[0], color='#5975a4', saturation=1) sns.histplot(x="who", data=titanic, ax=ax[1], color='#5975a4', alpha=1) plt.show()   PS: By default, a countplot uses only 80% of the width, while a histogram uses the full width. If desired, the histogram bars can be shrunk, e.g. sns.histplot(..., shrink=0.8), to get the same width as the countplot.
Question: I often forget to prefix formatted strings with "f". A buggy example: text = "results is {result}" Where it should be text = f"results is {result}" I make this error A LOT; my IDE don't report it, and the program runs without exceptions. I thought maybe to scan my source code for quoted strings, check for {,} characters, and search if it's missing a prefix "f" literal; But it's better, i guess, to use a parser? Or maybe someone did it already?

Answer: The problem is that text = "results is {result}" is a valid template string, so you can later use it in your program like: >>> text.format(result=1) 'results is 1' >>> text.format(result=3) 'results is 3'  What you can achieve is just checking if an f-string does indeed use variables inside, like pylint and flake8 already do. For what you seek, however, there is something going on with PyLint: this issue is 3 years old, but it is exactly what you need, and recently - three days ago - an user submitted a pull request that is currently WIP, and should resolve your problem.