eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Is it possible to use an integer key with TypedDict (similar to dict?). Trying a simple example:  from typing import TypedDict class Moves(TypedDict): 0: int=1 1: int=2  Throws: SyntaxError: illegal target for annotation It seems as though only Mapping[str, int] is supported but I wanted to confirm. It wasn't specifically stated in the Pep docs.

Answer: The intent of TypedDict is explicit in the PEP's abstract (emphasis added):  This PEP proposes a type constructor typing.TypedDict to support the use case where a dictionary object has a specific set of string keys, each with a value of a specific type.  and given the intended use cases are all annotatable in class syntax, implicitly applies only to dicts keyed by strings that constitute valid identifiers (things you could use as attribute or keyword argument names), not even strings in general. So as intended, int keys aren't a thing, this is just for enabling a class that uses dict-like syntax to access the "attributes" rather than attribute access syntax. While the alternative, backwards compatible syntax, allowed for compatibility with pre-3.6 Python, allows this (as well as allowing strings that aren't valid Python identifiers), e.g.: Moves = TypedDict('Moves', {0: int, 1: int})  you could only construct it with dict literals (e.g. Moves({0: 123, 1: 456})) because the cleaner keyword syntax like Moves(0=123, 1=456) doesn't work. And even though that technically works at runtime (it's all just dicts under the hood after all), the actual type-checkers that validate your type correctness may not support it (because the intent and documented use exclusively handles strings that constitute valid identifiers). Point is, don't do this. For the simple case you're describing here (consecutive integer integer "keys" starting from zero, where each position has independent meaning, where they may or may not differ by type), you really just want a tuple anyway: Moves = typing.Tuple[int, int] # Could be [int, str] if index 1 should be a string  would be used for annotations the same way, and your actual point of use in the code would just be normal tuple syntax (return 1, 2). If you really want to be able to use the name Moves when creating instances, on 3.9+ you could use PEP 585 to do (no import required): Moves = tuple[int, int]  allowing you to write: return Moves((1, 2))  when you want to make an "instance" of it. No runtime checking is involved (it's roughly equivalent to running tuple((1, 2)) at runtime), but static type-checkers should understand the intent.
Question: Python version: 3.9.1 I trying to write bot that send requests and it work perfectly fine, the only issue that i have is when i trying to use web debugging programs such as Charles 4.6.1 or Fiddler Everywhere. When I open it to see bot traffic and response form server it crash showing me this error: (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1124)')))  I used to have this issue and I was able to fix it by simply adding verify=False to my request, but right now it does not work.

Answer: I had the same problem. It's a bug in urllib3. You have to specify your proxy in the request, and change the 'https' value to 'http'. My example: proxies = {'https': 'http://127.0.0.1:8888'} request = r.get('https://www.example.net', verify=False, proxies=proxies)
Question: I am trying to run a service that uses simple transformers Roberta model to do classification. the inferencing script/function itself is working as expected when tested. when i include that with fast api its shutting down the server. uvicorn==0.11.8 fastapi==0.61.1 simpletransformers==0.51.6 cmd : uvicorn --host 0.0.0.0 --port 5000 src.main:app   @app.get("/article_classify") def classification(text:str): """function to classify article using a deep learning model. Returns: [type]: [description] """ _,_,result = inference(text) return result  error : INFO: Started server process [8262] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit) INFO: 127.0.0.1:36454 - "GET / HTTP/1.1" 200 OK INFO: 127.0.0.1:36454 - "GET /favicon.ico HTTP/1.1" 404 Not Found INFO: 127.0.0.1:36454 - "GET /docs HTTP/1.1" 200 OK INFO: 127.0.0.1:36454 - "GET /openapi.json HTTP/1.1" 200 OK before 100%|████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 17.85it/s] INFO: Shutting down INFO: Finished server process [8262]  inferencing script : model_name = "checkpoint-3380-epoch-20" model = MultiLabelClassificationModel("roberta","src/outputs/"+model_name) def inference(input_text,model_name="checkpoint-3380-epoch-20"): """Function to run inverence on one sample text""" #model = MultiLabelClassificationModel("roberta","src/outputs/"+model_name) all_tags =[] if isinstance(input_text,str): print("before") result ,output = model.predict([input_text]) print(result) tags=[] for idx,each in enumerate(result[0]): if each==1: tags.append(classes[idx]) all_tags.append(tags) elif isinstance(input_text,list): result ,output = model.predict(input_text) tags=[] for res in result : for idx,each in enumerate(res): if each==1: tags.append(classes[idx]) all_tags.append(tags) return result,output,all_tags  update: tried with flask and the service is working but when adding uvicorn on top of flask its getting stuck in a loop of restart.

Answer: I have solved this issue by starting a process pool using multiprocessing explicitly. from multiprocessing import set_start_method from multiprocessing import Process, Manager try: set_start_method('spawn') except RuntimeError: pass @app.get("/article_classify") def classification(text:str): """function to classify article using a deep learning model. Returns: [type]: [description] """ manager = Manager() return_result = manager.dict() # as the inference is failing p = Process(target = inference,args=(text,return_result,)) p.start() p.join() # print(return_result) result = return_result['all_tags'] return result
Question: Let's say I have a very simple logging decorator: from functools import wraps def my_decorator(func): @wraps(func) def wrapper(*args, **kwargs): print(f"{func.__name__} ran with args: {args}, and kwargs: {kwargs}") result = func(*args, **kwargs) return result return wrapper  I can add this decorator to every pytest unit test individually: @my_decorator def test_one(): assert True @my_decorator def test_two(): assert 1  How can I automatically add this decorator to every single pytest unit test so I don't have to add it manually? What if I want to add it to every unit test in a file? Or in a module? My use case is to wrap every test function with a SQL profiler, so inefficient ORM code raises an error. Using a pytest fixture should work, but I have thousands of tests so it would be nice to apply the wrapper automatically instead of adding the fixture to every single test. Additionally, there may be a module or two I don't want to profile so being able to opt-in or opt-out an entire file or module would be helpful.

Answer: Provided you can move the logic into a fixture, as stated in the question, you can just use an auto-use fixture defined in the top-level conftest.py. To add the possibility to opt out for some tests, you can define a marker that will be added to the tests that should not use the fixture, and check that marker in the fixture, e.g. something like this: conftest.py import pytest def pytest_configure(config): config.addinivalue_line( "markers", "no_profiling: mark test to not use sql profiling" ) @pytest.fixture(autouse=True) def sql_profiling(request): if not request.node.get_closest_marker("no_profiling"): # do the profiling yield  test.py import pytest def test1(): pass # will use profiling @pytest.mark.no_profiling def test2(): pass # will not use profiling  As pointed out by @hoefling, you could also disable the fixture for a whole module by adding: pytestmark = pytest.mark.no_profiling  in the module. That will add the marker to all contained tests.
Question: I'm trying to parse all the company names from this webpage. There are around 2431 companies in there. However, the way I've tried below can fetches me 1000 results. This is what I can see about the number of results in response while going through dev tools: hitsPerPage: 1000 index: "YCCompany_production" nbHits: 2431 <------------------------ nbPages: 1 page: 0   How can I get the rest of the results using requests?  I've tried so far: import requests url = 'https://45bwzj1sgc-dsn.algolia.net/1/indexes/*/queries?' params = { 'x-algolia-agent': 'Algolia for JavaScript (3.35.1); Browser; JS Helper (3.1.0)', 'x-algolia-application-id': '45BWZJ1SGC', 'x-algolia-api-key': 'NDYzYmNmMTRjYzU4MDE0ZWY0MTVmMTNiYzcwYzMyODFlMjQxMWI5YmZkMjEwMDAxMzE0OTZhZGZkNDNkYWZjMHJlc3RyaWN0SW5kaWNlcz0lNUIlMjJZQ0NvbXBhbnlfcHJvZHVjdGlvbiUyMiU1RCZ0YWdGaWx0ZXJzPSU1QiUyMiUyMiU1RCZhbmFseXRpY3NUYWdzPSU1QiUyMnljZGMlMjIlNUQ=' } payload = {"requests":[{"indexName":"YCCompany_production","params":"hitsPerPage=1000&query=&page=0&facets=%5B%22top100%22%2C%22isHiring%22%2C%22nonprofit%22%2C%22batch%22%2C%22industries%22%2C%22subindustry%22%2C%22status%22%2C%22regions%22%5D&tagFilters="}]} with requests.Session() as s: s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36' r = s.post(url,params=params,json=payload) print(len(r.json()['results'][0]['hits'])) 

Answer: As a workaround you can simulate search using alphabet as a search pattern. Using code below you will get all 2431 companies as dictionary with ID as a key and full company data dictionary as a value. import requests import string params = { 'x-algolia-agent': 'Algolia for JavaScript (3.35.1); Browser; JS Helper (3.1.0)', 'x-algolia-application-id': '45BWZJ1SGC', 'x-algolia-api-key': 'NDYzYmNmMTRjYzU4MDE0ZWY0MTVmMTNiYzcwYzMyODFlMjQxMWI5YmZkMjEwMDAxMzE0OTZhZGZkNDNkYWZjMHJl' 'c3RyaWN0SW5kaWNlcz0lNUIlMjJZQ0NvbXBhbnlfcHJvZHVjdGlvbiUyMiU1RCZ0YWdGaWx0ZXJzPSU1QiUyMiUy' 'MiU1RCZhbmFseXRpY3NUYWdzPSU1QiUyMnljZGMlMjIlNUQ=' } url = 'https://45bwzj1sgc-dsn.algolia.net/1/indexes/*/queries' result = dict() for letter in string.ascii_lowercase: print(letter) payload = { "requests": [{ "indexName": "YCCompany_production", "params": "hitsPerPage=1000&query=" + letter + "&page=0&facets=%5B%22top100%22%2C%22isHiring%22%2C%22nonprofit%22%2C%22batch%22%2C%22industries%22%2C%22subindustry%22%2C%22status%22%2C%22regions%22%5D&tagFilters=" }] } r = requests.post(url, params=params, json=payload) result.update({h['id']: h for h in r.json()['results'][0]['hits']}) print(len(result))
Question: I am getting the following error for the code below, please help:  from delta.tables import * ModuleNotFoundError: No module named 'delta.tables' INFO SparkContext: Invoking stop() from shutdown hook  Here is the code: '''  from pyspark.sql import * if __name__ == "__main__": spark = SparkSession \ .builder \ .appName("DeltaLake") \ .config("spark.jars", "delta-core_2.12-0.7.0") \ .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \ .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \ .getOrCreate() from delta.tables import * data = spark.range(0, 5) data.printSchema()  ''' An online search suggesting verifying the scala version to delta core jar version. Here is the scala & Jar versions "delta-core_2.12-0.7.0" "Using Scala version 2.12.10, Java HotSpot(TM) 64-Bit Server VM, 1.8.0_221"

Answer: According to delta package documentation, there is a python file named tables. You should clone the repository and copy the delta folder under python/delta to your site packages path (i.e. ..\python37\Lib\site-packages). then restart python and your code runs without the error. I am using Python3.5.3, pyspark==3.0.1,
Question: I'm trying to migrate my entire workflow from eclipse and jupyter notebook all over to VS Code. I installed the python extension, which should come with Intellisense, but it only works partly. I get suggestions after typing a period (.), but don't get any information on parameters nor documentation when hovering over with my mouse. Thank you so much for your help and have a wonderful new year! P.S If anyone has any experience with using anaconda environments with VS Code, that would be greatly appreciated as well as I running into some problems with it recognizing the libraries.   Also you can see here that when I manually activate IntelliSense, it doesn't recognize that it's in a method.  Sorry for the long string of edits, but I discovered that when typing print in a regular Python file, it works, but not in a Jupyter Notebook file. Also, it still doesn't work for numpy. Thanks for the help everyone. 

Answer: You could use the shortcut key "Ctrl+Space" to open the suggested options:  In addition, it is recommended that you use the extension "Pylance", which works better with the extension "Python".  Update: Currently in VSCode, the "IntelliSense" document content is provided by the Python language service, which is mainly for Python files (".py" files call this function), while in Jupyter, the "IntelliSense" used by the ".ipynb" file comes from the extension "Jupyter". You could refer to the content of this link to use VS code insiders, and its notebook editor has better intellisense. In VS code insiders:
Question: I have a geopandas dataframe, which consists of the region name(District), the geometry column, and the amount column. My goal is to plot a choropleth map using the method mentioned below https://plotly.com/python/choropleth-maps/#using-geopandas-data-frames Here’s a snippet of my dataframe  I also checked that my columns were in the right format/type.  And here's the code I used to plot the map fig = px.choropleth(merged, geojson=merged.geometry, locations=merged.index, color="Amount") fig.update_geos(fitbounds="locations", visible=False) fig.show()  It produced the below figure  which is obviously not the right figure. For some reasons, it doesn't show the map, instead it shows a line and when I zoom in, I am able to see the map but it has lines running through it. Like this  Has anyone ran into a similar problem? If so how were you able to resolve it? The Plotly version I am using is 4.7.0. I have tried upgrading to a most recent version but it still didn’t work. Any help is greatly appreciated. Please find my code and the data on my github.

Answer: I'll give you the answer to @tgrandje's comment that solved the problem. Thanks to @Poopah and @tgrandje for the opportunity to raise the answer. import pandas as pd import plotly.express as px import geopandas as gpd import pyproj # reading in the shapefile fp = "./data/" map_df = gpd.read_file(fp) map_df.to_crs(pyproj.CRS.from_epsg(4326), inplace=True) df = pd.read_csv("./data/loans_amount.csv") # join the geodataframe with the cleaned up csv dataframe merged = map_df.set_index('District').join(df.set_index('District')) #merged = merged.reset_index() merged.head() fig = px.choropleth(merged, geojson=merged.geometry, locations=merged.index, color="Amount") fig.update_geos(fitbounds="locations", visible=False) fig.show()
Question: I installed pyenv-win on my windows machine. It works fine in the command line. I can install python versions, set them as global etc. But My VS Code doesn't see them. It only sees one python interpreter I installed a long time ago when I wasn't using pyenv yet. VScode:  pyenv: C:\Users\jbron\cmder λ pyenv versions 3.7.0 * 3.8.0 (set by C:\Users\jbron\.pyenv\pyenv-win\version)  Why is it not finding my pyenv interpreters? I don't have problems like that on my Linux machines

Answer: It is recommended that you try the following:  Please check whether the Python environment variable contains your installed Python path:   Please reopen VSCode after installation:    Update: The environment variable path of "pyenv" I use is: (Under this path, we can find Python 3.6.7 downloaded by pyenv)   We can see the storage location where it downloaded Python 3.6.7:  C:\Users\...\.pyenv\pyenv-win\install_cache\python-3.6.7-amd64-webinstall.exe  Double-click to install:
Question: How to in python 3.9 implement the functionality of calling the async functions with different parameters, by scheduled periods? The functionality should be working on any OS (Linux, Windows, Mac) I have a function fetchOHLCV which downloads market data from exchanges. The function has two input parameters - pair, timeframe. Depending on the pair and timeframe values the function download data from exchanges and stores them in DB. The goal - call this function with different periods with different parameters. 1) fetchOHLCV(pair ='EUR/USD', timeframe="1m") - each minute 2) fetchOHLCV(pair ='EUR/USD', timeframe="1h") - each new hour. 3) fetchOHLCV(pair ='EUR/USD', timeframe="1d") - each new day 4) fetchOHLCV(pair ='EUR/USD', timeframe="1w") - each new week  At this moment I don't have experience working with scheduling in python and I don't know which libraries will optimal for my task and I'm interested in the best practices of implementing similar tasks.

Answer: As a ready-to-use solution, you could use aiocron library. If you are not familiar with cron expression syntax, you can use this online editor to check. Example (this will work on any OS): import asyncio from datetime import datetime import aiocron async def foo(param): print(datetime.now().time(), param) async def main(): cron_min = aiocron.crontab('*/1 * * * *', func=foo, args=("At every minute",), start=True) cron_hour = aiocron.crontab('0 */1 * * *', func=foo, args=("At minute 0 past every hour.",), start=True) cron_day = aiocron.crontab('0 9 */1 * *', func=foo, args=("At 09:00 on every day-of-month",), start=True) cron_week = aiocron.crontab('0 9 * * Mon', func=foo, args=("At 09:00 on every Monday",), start=True) while True: await asyncio.sleep(1) asyncio.run(main())  Output: 15:26:00.003226 At every minute 15:27:00.002642 At every minute ...
Question: How do I run a simple loop in VS Code's python debug console? When I try to enter the following: for el in dataset:  It gives me the error below. I seem to be able to enter variable names, but not multi-line commands like I can in the normal python REPL. Traceback (most recent call last): File "/home/tensorflow/.local/lib/python3.6/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_vars.py", line 416, in evaluate_expression compiled = compile(_expression_to_evaluate(expression), '<string>', 'eval') File "<string>", line 1 for el in dataset: ^ SyntaxError: invalid syntax During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/tensorflow/.local/lib/python3.6/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 969, in internal_evaluate_expression_json pydevd_vars.evaluate_expression(py_db, frame, expression, is_exec=True) File "/home/tensorflow/.local/lib/python3.6/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_vars.py", line 368, in new_func return _run_with_unblock_threads(original_func, py_db, curr_thread, frame, expression, is_exec) File "/home/tensorflow/.local/lib/python3.6/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_vars.py", line 336, in _run_with_unblock_threads return _run_with_interrupt_thread(original_func, py_db, curr_thread, frame, expression, is_exec) File "/home/tensorflow/.local/lib/python3.6/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_vars.py", line 307, in _run_with_interrupt_thread return original_func(py_db, frame, expression, is_exec) File "/home/tensorflow/.local/lib/python3.6/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_vars.py", line 418, in evaluate_expression Exec(_expression_to_evaluate(expression), updated_globals, frame.f_locals) File "/home/tensorflow/.local/lib/python3.6/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec exec(exp, global_vars, local_vars) File "<string>", line 1 for el in dataset: ^ SyntaxError: unexpected EOF while parsing 

Answer: You have 2 options:  Write the command in a new editor window, then simply copy and paste the code in the debug console and press Enter Write the command directly in the debug console. When you want to enter a new line, press Shift+Enter. When the command is complete, execute with Enter
Question: My code works fine for English text, but doesn't work for for Russian search_text. How can I fix it? Error text  UnicodeEncodeError: 'latin-1' codec can't encode characters in position 41-46: Body ('Москва') is not valid Latin-1. Use body.encode('utf-8') if you want to send it encoded in UTF-8.  My code import requests # search_text = "London" # OK: for english text search_text = "Москва" # ERROR: 'latin-1' codec can't encode characters in position 41-46: Body ('Москва') headers = { 'cookie': 'bci=6040686626671285074; _statid=a741e249-8adb-4c9a-8344-6e7e8360700a; viewport=762; _hd=h; tmr_lvid=ea50ffe34e269b16d061756e9a17b263; tmr_lvidTS=1609852383671; AUTHCODE=VCmGBS9d9sIxDnxN-hzApvPxPoLNADWCZLYyW8JOTcolv2dJjwH7ALYd8dNP9ljxZZuLvoKsDXgozEUt-PjSwXYEDt4syizx1I2LS58gb49kCFae-5uIap--mtLsff2ZqGbFqK5r7buboZ0_3; JSESSIONID=adca48748b8f0c58a926f5e4948f42c0c0aa9463798a9240.1f3566ed; LASTSRV=ok.ru; msg_conf=2468555756792551; TZ=6; _flashVersion=0; CDN=; nbp=; tmr_detect=0%7C1609852395541; cudr=0; klos=0; tmr_reqNum=4; TZD=6.200; TD=200', } data = '''{\n "id": 24,\n "parameters": {\n "query": "''' + search_text + '''"\n }\n}''' response = requests.post('https://ok.ru/web-api/v2/search/suggestCities', headers=headers, data=data) json_data = response.json() print(json_data['result'][0]['id'])  I tried city_name = city_name.encode('utf-8')  but received TypeError: must be str, not bytes

Answer: Try adding this after the line where you create the data variable before you post the request data=data.encode() #will produce bytes object encoded with utf-8
Question: Imagine we have some giant legacy code base with a lot of files with ignored Mypy warnings: def foobar(): x = some_external_class.some_method()[0] # type: ignore[ignore-some-mypy-warning]  Time to go... Some parts of code were changed. Some parts of code is still the same. How to check every "ignore" comment to know: will I get an error if I remove it? Desired output: Checked 100500 files! You do not need "ignore" comments anymore in the following files: - spam.py:534 - eggs.py:31 - eggs.py:250  Are there any existing tools to achieve this? Any ideas about custom scripts? The only idea that I have:  Write a script that will find and remember a file and a line of every Mypy comment. Find and remove all Mypy comments. Run Mypy check → store results. Compare the Mypy check errors lines with a stored old lines. Find a difference: if a comment was removed, but Mypy does not complain now about that line, then the comment must be removed. 

Answer: From mypy documentation:  --warn-unused-ignores This flag will make mypy report an error whenever your code uses a # type: ignore comment on a line that is not actually generating an error message.
Question: Float32 vs float32? What is the purpose of uppercase vs lowercase dtypes in Pandas? Uppercase seems more error prone: TypeError: object cannot be converted to a FloatingDtype. dtype = { 'doom_float64': 'Float64' , 'radiance_float32': 'Float32' , 'temperature_float': 'float' , 'moonday_int64': 'Int64' , 'month_int32': 'Int32' , 'color_uint8': 'UInt8' , 'shape_int': 'int' , 'weekday_object': 'object' , 'hour_object': 'string' , 'kingdom_category': 'category' }  >>> df.dtypes doom_float64 Float64 radiance_float32 Float32 temperature_float float64 weekday_object object hour_object string moonday_int64 Int64 month_int32 Int32 color_uint8 UInt8 shape_int int64 kingdom_category category dtype: object  Pandas v1.2.0

Answer: Yes, see here for example.  pandas can represent integer data with possibly missing values using arrays.IntegerArray. This is an extension types implemented within pandas. Or the string alias "Int64" (note the capital "I", to differentiate from NumPy’s 'int64' dtype:  Capitalized types are pandas types, while uncapitalized types are numpy types. One feature of pandas types is the ability to support nan, which is not a standard IEEE for non floats.