eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Is there a way to set a pydantic model from a list? I tried this and it didn't work for me. If it's not possible with pydantic, what is the best way to do this if I still need type validation and conversion, constraints, etc.? Order is important here. from pydantic import BaseModel from datetime import date class User(BaseModel): id: int name = 'John Doe' sex: str money: float = None dt: date data = [1, 'Tike Myson', 'male', None, '2022-01-20'] user = User(*data) >>> TypeError: __init__() takes exactly 1 positional argument (6 given) 

Answer: I partially answered it here: Initialize FastAPI BaseModel using non-keywords arguments (a.k.a *args) but I'll give here more dynamic options. Option 1: use the order of the attributes Your case has the problem that Pydantic does not maintain the order of all fields (depends at least on whether you set the type). If you specify the type of name then this works: from pydantic import BaseModel from datetime import date class User(BaseModel): id: int name: str = 'John Doe' sex: str money: float = None dt: date def __init__(self, *args): # Get a "list" of field names (or key view) field_names = self.__fields__.keys() # Combine the field names and args to a dict # using the positions. kwargs = dict(zip(field_names, args)) super().__init__(**kwargs) data = [1, 'Tike Myson', 'male', None, '2022-01-20'] user = User(*data)  Option 2: set the order as class variable This has the downside of not being as dynamic but does not have the problem of the order being undesirable. from pydantic import BaseModel from datetime import date class User(BaseModel): id: int name = 'John Doe' sex: str money: float = None dt: date field_names: ClassVar = ('id', 'name', 'sex', 'money', 'dt') def __init__(self, *args): # Combine the field names and args to a dict # using the positions. kwargs = dict(zip(self.field_names, args)) super().__init__(**kwargs) data = [1, 'Tike Myson', 'male', None, '2022-01-20'] user = User(*data)  Option 3: just hard-code them in the __init__ This is similar to option 2 but a bit simple (and less reusable). from pydantic import BaseModel from datetime import date from typing import ClassVar class User(BaseModel): id: int name = 'John Doe' sex: str money: float = None dt: date def __init__(self, id, name, sex, money, dt): super().__init__(id=id, name=name, sex=sex, money=money, dt=dt) data = [1, 'Tike Myson', 'male', None, '2022-01-20'] user = User(*data)  Option 4: Without overriding One more solution but this does not require overriding the __init__. However, this requires more code when creating an instance: from pydantic import BaseModel from datetime import date class User(BaseModel): id: int name = 'John Doe' sex: str money: float = None dt: date data = [1, 'Tike Myson', 'male', None, '2022-01-20'] # Combine the field names and data field_names = ['id', 'name', 'sex', 'money', 'dt'] kwargs = dict(zip(field_names, data)) # Create an instance of User from a dict user = User(**kwargs)  Note for options 1, 2 & 3 If you have multiple places where you need this, create a base class that has the __init__ overridden and subclass that.
Question: Overview I am using a proxy network and want to configure it with Selenium on Python. I have seen many post use the HOST:PORT method, but proxy networks uses the "URL method" of http://USER:PASSWORD@PROXY:PORT SeleniumWire I found SeleniumWire to be a way to connect the "URL method" of proxy networks to a Selenium Scraper. See basic SeleniumWire configuration: from seleniumwire import webdriver options = { 'proxy': { 'http': 'http://USER:PASSWORD@PROXY:PORT', 'https': 'http://USER:PASSWORD@PROXY:PORT' }, } driver = webdriver.Chrome(seleniumwire_options=options) driver.get("https://some_url.com")  This correctly adds and cycles a proxy to the driver, however on many websites the scraper is quickly blocked by CloudFlare. This blocking is something that does not happen when running on Local IP. After searching through SeleniumWire's GitHub Repository Issues, I found that this is caused by TLS fingerprinting and that there is no current solution to this issue. Selenium Options I tried to configure proxies the conventional selenium way: from selenium import webdriver options = webdriver.ChromeOptions() options.add_argument("--proxy-server=http://USER:PASSWORD@PROXY:PORT") driver = webdriver.Chrome(options=options) driver.get("https://some_url.com")  A browser instance does open but fails because of a network error. Browser instance does not load in established URL.  Docker Configuration The end result of this configuration would be running python code within a docker container that is within a Lambda function. Don't know whether or not that introduces a new level of abstraction or not. Summary What other resources can I use to correctly configure my Selenium scraper to use the "URL method" of IP cycling? Versions  python 3.9 selenium 3.141.0 docker 20.10.11  Support Tickets Github: https://github.com/SeleniumHQ/selenium/issues/10605 ChromeDriver: https://bugs.chromium.org/p/chromedriver/issues/detail?id=4118

Answer: Selenium Extension: A proxy network, or "URL" proxy, can be configured with Selenium as an extension. Create the following JS script and JSON file: JS script ("background.js") var config = { mode: "fixed_servers", rules: { singleProxy: { scheme: "http", host: "<PROXY>", port: parseInt(<PORT>) }, bypassList: ["foobar.com"] } }; chrome.proxy.settings.set({value: config, scope: "regular"}, function() {}); function callbackFn(details) { return { authCredentials: { username: "<USER>", password: "<PASSWORD>" } }; } chrome.webRequest.onAuthRequired.addListener( callbackFn, {urls: ["<all_urls>"]}, ['blocking'] );  JSON File ("manifest.json") { "version": "1.0.0", "manifest_version": 2, "name": "Chrome Proxy", "permissions": [ "proxy", "tabs", "unlimitedStorage", "storage", "<all_urls>", "webRequest", "webRequestBlocking" ], "background": { "scripts": ["background.js"] }, "minimum_chrome_version":"22.0.0" }   Zip background.js and manifest.json as proxy.zip and write the following: from selenium import webdriver options = webdriver.ChromeOptions() options.add_extension("proxy.zip") driver = webdriver.Chrome(options=options) driver.get("https://whatismyipaddress.com/")
Question: I wrote a python script to download a list of YouTube URLs, and I want to change the output folder by the subject I'm downlaoding. For example, When I'm downloading a playlist, I want the videos in this playlist be downloaded into a folder named by the current playlist. But if it's a channel, the videos inside it should go into a folder named bt it's uploader. How do I know the URL I'm downloading is a playlist or a channel? Since the options are passed before the download starts, I can't find a way to do this. Here is my code: import sys import yt_dlp URLS = [ 'playlist_url', 'channel_url', ] dl_ops = { 'outtmpl': 'd:/YouTube/%(uploader)s/%(title)s.%(ext)s' } retry_count = 0 def download_video(urls): try: with yt_dlp.YoutubeDL(dl_ops) as ydl: ydl.download(urls) except KeyboardInterrupt: print('Interruptted by user') sys.exit() except Exception as e: print(e) global retry_count if retry_count == 50: print('Retry count exceeded') sys.exit() retry_count += 1 download_video(urls) if __name__ == '__main__': download_video(URLS) 

Answer: dl_ops = { 'outtmpl': 'd:/YouTube/%(uploader)s/%(title)s.%(ext)s' }  You can see that in this line you specify a path to be D:/YouTube/ {uploader name} / {title of the video} . {extension that you specified}. In order to change all the videos to go into the same folder you just need to remove the "/" after the uploader so your videos don't have a seperate folder for each of the uploaders. So if you want to have file name just as video name you will do something like this: 'outtmpl': 'D:/YouTube/%(title)s.%(ext)s'  So for a video named "Youtube vlog part 321" the file path will be: D:/YouTube/Youtube vlog part 321.mp4  or any other format that you choose (mp3, wav, avi, mkv...).
Question: I'm trying to hide the entire schemas section of the FastAPI generated swagger docs. I've checked the docs and tried this but the schema section still shows.  @Schema(hidden=True) class theSchema(BaseModel): category: str  How do I omit one particular schema or the entire schemas section from the returned swagger docs. docExpansion does not appear to work either. What am I missing? app = FastAPI( openapi_tags=tags_metadata, title="Documentation", description="API endpoints", version="0.1", docExpansion="None" ) 

Answer: swagger has the UI parameter "defaultModelsExpandDepth" for controlling models' view in the schema section. This parameter can be forwarded using "swagger_ui_parameters" parameter on FastApi initialization. app = FastAPI(swagger_ui_parameters={"defaultModelsExpandDepth": -1})  Values:  -1: schema section hidden 0: schema section is closed 1: schema section is open (default)  More options can be found here: https://swagger.io/docs/open-source-tools/swagger-ui/usage/configuration/
Question: I am running coroutines of 'workers' whose job it is to wait 5s, get values from an asyncio.Queue() and print them out continually. q = asyncio.Queue() def worker(): while True: await asyncio.sleep(5) i = await q.get() print(i) q.task_done() async def main(q): workers = [asyncio.create_task(worker()) for n in range(10)] await asyncio.gather(*workers) if __name__ == "__main__": asyncio.run(main())  I would like to be able to interact with the queue through http requests using FastAPI. For example POST requests that would 'put' items in the queue for the workers to print. I'm unsure how I can run the coroutines of the workers concurrently with FastAPI to achieve this effect. Uvicorn has its own event loop I believe and my attempts to use asyncio methods have been unsuccessful. The router would look something like this I think. @app.post("/") async def put_queue(data:str): return q.put(data)  And I'm hoping there's something that would have an effect like this: await asyncio.gather(main(),{FastApi() app run}) 

Answer: One option would be to add a task that wraps your main coroutine in a on startup event import asyncio @app.on_event("startup") async def startup_event(): asyncio.create_task(main())  This would schedule your main coroutine before the app has been fully started. Important here is that you don't await the created task as it would basically block startup_event forever
Question: i have a list of strings with some strings being the special characters what would be the approach to exclude them in the resultant list list = ['ben','kenny',',','=','Sean',100,'tag242'] expected output = ['ben','kenny','Sean',100,'tag242']  please guide me with the approach to achieve the same. Thanks

Answer: The string module has a list of punctuation marks that you can use and exclude from your list of words: import string punctuations = list(string.punctuation) input_list = ['ben','kenny',',','=','Sean',100,'tag242'] output = [x for x in input_list if x not in punctuations] print(output)  Output: ['ben', 'kenny', 'Sean', 100, 'tag242']  This list of punctuation marks includes the following characters: ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~']
Question: I'm having difficulty specifying python path containing modules/packages in another directory or even folder of the same project. When I try to import I get the error: ModuleNotFoundError: No module named 'perception' In Spyder this is simply done using the UI to select an additional pythonpath that python will look in, but I am unable to do so in VSC. Note I have tried following other answers on editing the settings.json files, and .env files but the problem still persists. The only solution I have is to use sys.path.append() in every script which is not what I am looking for. As an example my settings.json file is: { "terminal.integrated.env.osx": { "PYTHONPATH": "pathtoprojectfolder" }, "python.envFile": "${workspaceFolder}/.env", "jupyter.interactiveWindowMode": "perFile", "python.terminal.executeInFileDir": true, "terminal.integrated.inheritEnv": true, "jupyter.themeMatplotlibPlots": true, "window.zoomLevel": 2, "python.condaPath": "path_to_conda_python", "python.defaultInterpreterPath": "path_to_conda_python", } 

Answer: Instructions for MacOS only. Adding .env files and navigating the setting json files is not as intuitive and simple as adding an additional python path in Spyder. However, this worked for me on VSC:  create a .env file in the project folder. add the full path that you want to add to PYTHONPATH as such: PYTHONPATH=/Users/../projectFolder:/Users/.../AnotherProjectFolder  ** crucially you must use the appropriate separator between paths, on Mac you must use ':', otherwise the paths will not be added.  go to to Code->preferences->settings and search for "terminal.integrated.osx". click edit "settings.json"  add to the json settings "python.envFile": "${workspaceFolder}/.env",   As an example you might to have settings.json showing the following: { "python.envFile": "${workspaceFolder}/.env", "jupyter.interactiveWindowMode": "perFile", "python.terminal.executeInFileDir": true, }  Quite an involved process to just add a path, but I like the ability to run Jupyter notebooks in VSC which Spyder doesn't have in the standalone installers yet. This page has the information required: https://code.visualstudio.com/docs/python/environments#_environment-variable-definitions-file
Question: I'm trying to accelerate my model's performance by converting it to OnnxRuntime. However, I'm getting weird results, when trying to measure inference time. While running only 1 iteration OnnxRuntime's CPUExecutionProvider greatly outperforms OpenVINOExecutionProvider:  CPUExecutionProvider - 0.72 seconds OpenVINOExecutionProvider - 4.47 seconds  But if I run let's say 5 iterations the result is different:  CPUExecutionProvider - 3.83 seconds OpenVINOExecutionProvider - 14.13 seconds  And if I run 100 iterations, the result is drastically different:  CPUExecutionProvider - 74.19 seconds OpenVINOExecutionProvider - 46.96seconds  It seems to me, that the inference time of OpenVinoEP is not linear, but I don't understand why. So my questions are:  Why does OpenVINOExecutionProvider behave this way? What ExecutionProvider should I use?  The code is very basic: import onnxruntime as rt import numpy as np import time from tqdm import tqdm limit = 5 # MODEL device = 'CPU_FP32' model_file_path = 'road.onnx' image = np.random.rand(1, 3, 512, 512).astype(np.float32) # OnnxRuntime sess = rt.InferenceSession(model_file_path, providers=['CPUExecutionProvider'], provider_options=[{'device_type' : device}]) input_name = sess.get_inputs()[0].name start = time.time() for i in tqdm(range(limit)): out = sess.run(None, {input_name: image}) end = time.time() inference_time = end - start print(inference_time) # OnnxRuntime + OpenVinoEP sess = rt.InferenceSession(model_file_path, providers=['OpenVINOExecutionProvider'], provider_options=[{'device_type' : device}]) input_name = sess.get_inputs()[0].name start = time.time() for i in tqdm(range(limit)): out = sess.run(None, {input_name: image}) end = time.time() inference_time = end - start print(inference_time) 

Answer: The use of ONNX Runtime with OpenVINO Execution Provider enables the inferencing of ONNX models using ONNX Runtime API while the OpenVINO toolkit runs in the backend. This accelerates ONNX model's performance on the same hardware compared to generic acceleration on Intel® CPU, GPU, VPU and FPGA. Generally, CPU Execution Provider works best with small iteration since its intention is to keep the binary size small. Meanwhile, the OpenVINO Execution Provider is intended for Deep Learning inference on Intel CPUs, Intel integrated GPUs, and Intel® MovidiusTM Vision Processing Units (VPUs). This is why the OpenVINO Execution Provider outperforms the CPU Execution Provider during larger iterations. You should choose Execution Provider that would suffice your own requirements. If you going to execute complex DL with large iteration, then go for OpenVINO Execution Provider. For a simpler use case, where you need the binary size to be smaller with smaller iterations, you can choose the CPU Execution Provider instead. For more information, you may refer to this ONNX Runtime Performance Tuning
Question: I work on a program for two months. Today I suddenly got an error when connecting to the database while using mysql.connector. Interestingly, this error is not seen when running previous versions. import mysql.connector import pandas as pd mydb = mysql.connector.connect(host="localhost", user="root", password="*****", database="****") Q = f'SELECT * FROM table' df = pd.read_sql_query(Q, con=mydb) print(df)  but I get this error : Traceback (most recent call last): df = pd.read_sql_query(Q, con=mydb) File "g.v1.6\venv\lib\site-packages\pandas\io\sql.py", line 398, in read_sql_query pandas_sql = pandasSQL_builder(con) File "g.v1.6\venv\lib\site-packages\pandas\io\sql.py", line 750, in pandasSQL_builder sqlalchemy = import_optional_dependency("sqlalchemy") File "g.v1.6\venv\lib\site- packages\pandas\compat\_optional.py", line 129, in import_optional_dependency raise ImportError(msg) ImportError: Missing optional dependency 'SQLAlchemy'. Use pip or conda to install SQLAlchemy.  What has this got to do with SQLAlchemy??

Answer: I just ran into something similar. It looks like Pandas 1.4 was released on January 22, 2022: https://pandas.pydata.org/docs/dev/whatsnew/v1.4.0.html It has an "optional" dependency on SQLAlchemy, which is required to communicate with any database other than sqlite now, as the comment by snakecharmerb mentioned. Once I added that to my requirements and installed SQLAlchemy, it resolved my problem.
Question: As per API documentation https://docs.github.com/en/rest/reference/pulls#create-a-review-for-a-pull-request We can use a CURL to approve pull request i.e. curl -s -H "Authorization: token ghp_TOKEN" \ -X POST -d '{"event": "APPROVE"}' \ "https://api.github.com/repos/{owner}/{repo}/pulls/{pull_number}/reviews"  but I get this error: { "message": "Not Found", "documentation_url": "https://docs.github.com/rest" }  Although, the CURL is working fine for other APIs like: curl -s -H "Authorization: token ghp_TOKEN" \ -X POST -d '{"body": "some message"}' \ "https://api.github.com/repos/{owner}/{repo}/issues/{pull_number}/reviews"  I have tried everything to make this work. Can somebody please help me with this?

Answer: After bit of experiments, it has worked with API /repos/{owner}/{repo}/pulls/{pull_number}/reviews I must say that Github documentation is very poor that I have to spend almost 3 hours to figure this out. A small but proper CURL would have helped in a few seconds and would have saved my time. Anyway, leaving this solution on StackOverflow so that, this helps other people and saves their precious time. CURL: curl -s -H "Authorization: token ghp_BWuQzbiDANEvrQP9vZbqa5LHBAxxIzwi2gM7" \ -X POST -d '{"event":"APPROVE"}' \ "https://api.github.com/repos/tech-security/chatbot/pulls/4/reviews"  Python Code: import requests headers = { 'Authorization': 'token ghp_BWuQzbiDANEvrQP9vZbqa5LHBAxxIzwi2gM7', } data = '{"event":"APPROVE"}' response = requests.post('https://api.github.com/repos/tech-security/chatbot/pulls/4/reviews', headers=headers, data=data) print (response.json())  Note: above github token is dummy, so don't freak out please! :D
Question: Specializing the type of re.Pattern to re.Pattern[bytes], mypy correctly detects the type error: import re REGEX: re.Pattern[bytes] = re.compile(b"\xab.{2}") def check(pattern: str) -> bool: if str == "xyz": return REGEX.fullmatch(pattern) is not None return True print(check("abcd"))  Type mismatch detected: $ mypy ~/main.py /home/oren/main.py:5: error: Argument 1 to "fullmatch" of "Pattern" has incompatible type "str"; expected "bytes" Found 1 error in 1 file (checked 1 source file)  However, when I try to actually run the code I get a weird (?) message: $ python ~/main.py Traceback (most recent call last): File "/home/oren/main.py", line 2, in <module> REGEX: re.Pattern[bytes] = re.compile(b"\xab.{2}") TypeError: 'type' object is not subscriptable  How come the type annotation bothers Python?

Answer: The ability to specialize the generic re.Pattern and re.Match types using [str] or [bytes] was added in Python 3.9. It seems you are using an older Python version. For Python versions earlier than 3.8 the typing module provides a typing.re namespace which contains replacement types for this purpose. Since Python 3.8, they are directly available in the typing module and the typing.re namespace is deprecated (will be removed in Python 3.12). Reference: https://docs.python.org/3/library/typing.html#typing.Pattern Summary:  for Python <3.8, use typing.re.Pattern[bytes] for Python 3.8, use typing.Pattern[bytes] for Python 3.9+, use re.Pattern[bytes]
Question: I wrote a defaultdict subclass that can call default_factory with key as argument. from collections import defaultdict from typing import TypeVar, Any, Callable, Generic K = TypeVar('K') class keydefaultdict(defaultdict, Generic[K]): ''' Drop-in replacement for defaultdict accepting key as argument ''' default_factory: Callable[[], Any] | Callable[[K], Any] | None def __missing__(self, key: K) -> Any: if self.default_factory is None: raise KeyError(key) else: try: ret = self[key] = self.default_factory(key) except TypeError: # try no-key signature ret = self[key] = self.default_factory() # if failed, let the error propagate as usual return ret  mypy complains on the default_factory type hint:  incompatible type in assignment (expression has type "Callable[[], Any] | Callable[[K], Any] | None", base class "defaultdict" defined the type as "Optional[Callable[[], Any]]")  Is there any way to override the type? mypy complains also on this lines self.default_factory(...) - too many arguments (or too few arguments) and in places where this dict is instantiated (incompatible type): data = keydefaultdict(lambda key: [key])   Argument 1 to "keydefaultdict" has incompatible type "Callable[[Any], list[Any]]"; expected "Optional[Callable[[], Any]]" 

Answer: This is intended behavior, as what you're describing violates the Liskov substitution principle. See the mypy documentation for more details. Using defaultdict as a subclass is a bad idea for this reason. But, if you really want to get around this (not recommended), you can use # type: ignore[override], like so: default_factory: Callable[[], Any] | Callable[[K], Any] | None # type: ignore[override]  This is described in further detail in the mypy documentation if you want more information.
Question: Here is my code: import numpy as np import pandas as pd import seaborn as sns from sklearn.ensemble import IsolationForest data = pd.read_csv('marks1.csv', encoding='latin-1', on_bad_lines='skip', index_col=0, header=0 ) random_state = np.random.RandomState(42) model = IsolationForest(n_estimators=100, max_samples='auto', contamination=float(0.2) , random_state=random_state) model.fit(data[['Mark']]) random_state = np.random.RandomState(42) data['scores'] = model.decision_function(data[['Mark']]) data['anomaly_score'] = model.predict(data[['Mark']]) data[data['anomaly_score'] == -1].head()  Error:  C:\Program Files\Python39\lib\site-packages\sklearn\base.py:450: UserWarning: X does not have valid feature names, but IsolationForest was fitted with feature names warnings.warn( 

Answer: It depends on the version of sklearn you are using. In versions past 1.0, models have a feature_names attribute when trained with dataframes that integrates the column names. There was a bug in this version that threw an error when training with dataframes. https://github.com/scikit-learn/scikit-learn/issues/21577 I'm not up to date with the new best practices for this yet, so I cannot say definitively how it should be set up. But I just side stepped the issue in my code for now. To get around this, I convert my dataframes to a numpy array before training df.to_numpy()