eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I uploaded a Python app to Heroku. In my requirements.txt file I have a line for ipython==7.17.0 but Heroku seems unable to retrieve it, I don't understand why this might happen because I'm able to download that ipython version on my machine. The complete error thrown is: ERROR: Could not find a version that satisfies the requirement ipython==7.17.0 (from -r /tmp/build_76afe907/requirements.txt (line 32)) (from versions: 0.10, 0.10.1, 0.10.2, 0.11, 0.12, 0.12.1, 0.13, 0.13.1, 0.13.2, 1.0.0, 1.1.0, 1.2.0, 1.2.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 3.0.0, 3.1.0, 3.2.0, 3.2.1, 3.2.2, 3.2.3, 4.0.0b1, 4.0.0, 4.0.1, 4.0.2, 4.0.3, 4.1.0rc1, 4.1.0rc2, 4.1.0, 4.1.1, 4.1.2, 4.2.0, 4.2.1, 5.0.0b1, 5.0.0b2, 5.0.0b3, 5.0.0b4, 5.0.0rc1, 5.0.0, 5.1.0, 5.2.0, 5.2.1, 5.2.2, 5.3.0, 5.4.0, 5.4.1, 5.5.0, 5.6.0, 5.7.0, 5.8.0, 5.9.0, 5.10.0, 6.0.0rc1, 6.0.0, 6.1.0, 6.2.0, 6.2.1, 6.3.0, 6.3.1, 6.4.0, 6.5.0, 7.0.0b1, 7.0.0rc1, 7.0.0, 7.0.1, 7.1.0, 7.1.1, 7.2.0, 7.3.0, 7.4.0, 7.5.0, 7.6.0, 7.6.1, 7.7.0, 7.8.0, 7.9.0, 7.10.0, 7.10.1, 7.10.2, 7.11.0, 7.11.1, 7.12.0, 7.13.0, 7.14.0, 7.15.0, 7.16.0, 7.16.1) ERROR: No matching distribution found for ipython==7.17.0 (from -r /tmp/build_76afe907/requirements.txt (line 32)) 

Answer: The last allowed version is 7.16.1, that means you use Python 3.6. 7.17 requires Python 3.7+.
Question: This is the error: cv2.solvePnP(obj_points, image_points, mtx, dist) cv2.error: OpenCV(4.2.0) C:\projects\opencv-python\opencv\modules\calib3d\src\solvepnp.cpp:754: error: (-215:Assertion failed) ( (npoints >= 4) || (npoints == 3 && flags == SOLVEPNP_ITERATIVE && useExtrinsicGuess) ) && npoints == std::max(ipoints.checkVector(2, CV_32F), ipoints.checkVector(2, CV_64F)) in function 'cv::solvePnPGeneric'  And this is my code: mtx = np.load("./camera_params/mtx.npy") dist = np.load("./camera_params/dist.npy") obj_points = np.array([[0, 0, 0], [297, 0, 0], [297, 210, 0], [0, 210, 0]]) image_points = np.array([[416, 268], [422, 535], [826, 543], [829, 264]]) cv2.solvePnP(obj_points, image_points, mtx, dist)  I don't have any idea how to solve it. I tried to play with the arguments but it didn't help. If you know a way to solve this error it be very helpful.

Answer: I had this error very recently, and I resolved it by making the argument ndarrays floats instead of ints. You can do that in 2 ways: obj_points = np.array([[0.0, 0.0, 0.0], [297.0, 0.0, 0.0], [297.0, 210.0, 0.0], [0.0, 210.0, 0.0]]) image_points = np.array([[416.0, 268.0], [422.0, 535.0], [826.0, 543.0], [829.0, 264.0]])  obj_points = np.array([[0, 0, 0], [297, 0, 0], [297, 210, 0], [0, 210, 0]]) obj_points = obj_points.astype('float32') image_points = np.array([[416, 268], [422, 535], [826, 543], [829, 264]]) image_points = image_points.astype('float32')
Question: Can I make a default value in Pydantic if None is passed in the field? I have the following code, but it seems to me that the validator here only works on initialization of the model and not otherwise. My Code: class User(BaseModel): name: Optional[str] = '' password: Optional[str] = '' email: EmailStr @validator('name') def set_name(cls, name): return name or 'foo'  Problem Encountered: user = User(name=None, password='some_password', email='user@example.com') print("Name is ", user.name) # > 'Name is foo' user.name = None print("Name is ", user.name) # > 'Name is None'  Desired Output: user = User(name='some_name', password='some_password', email='user@example.com') user.name = None print("Name is ", user.name) # > 'Name is foo'  Any ideas on how I can obtain the desired output? I think having getters and setters will help in tackling the issue. However, I could not get them to work in a Pydantic model: Attempting to implement getters and setters: class User(BaseModel): name: Optional[str] = '' password: Optional[str] = '' email: EmailStr def get_password(self): return self.password def set_password(self, password): self.password = hash_password(password) password = property(get_password, set_password) user = User(name='some_name', password='some_password', email='user@example.com') # > RecursionError: maximum recursion depth exceeded  I also tried the property decorator: class User(BaseModel): name: Optional[str] = '' password: Optional[str] = '' email: EmailStr @property def password(self): return self._password @password.setter def password(self, password): pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto") self._password = pwd_context.hash(password) user = User(name='some_name', email='user@example.com') user.password = 'some_password' # > ValueError: "User" object has no field "password"  I also tried overwriting the init: class User(BaseModel): name: Optional[str] = "" password: Optional[str] = "" email: EmailStr def __init__(self, name, password, email): pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto") password = pwd_context.hash(password) super().__init__(name=name, password=password, email=email) user = User(name="some_name", password="some_password", email='user@example.com') print(user.password) # > AYylwSnbQgCHrl4uue6kO7yiuT20lazSzK7x # Works as expected user.password = "some_other_password" print(user.password) # > "some_other_password" # Does not work user.password = None print(user.password) # > None # Does not work either 

Answer: You need to enable validate_assignment option in model config: from typing import Optional from pydantic import BaseModel, validator class User(BaseModel): name: Optional[str] = '' password: Optional[str] = '' class Config: validate_assignment = True @validator('name') def set_name(cls, name): return name or 'foo' user = User(name=None, password='some_password', ) print("Name is ", user.name) user.name = None print("Name is ", user.name)  Name is foo Name is foo
Question: I'm using boto3 in a lambda function to write information into a dynamodb table. I get the error put_item() only accepts keyword arguments. Searching on the web, i found that this error may mean that I am not matching dynamodb partition key, but it seems to me that I am doing everything correctly. Can anyone help me find the error? Excuse me but this is my first time using aws. This is my lambda function import json, boto3 def updateDynamo(Item): dynamodb = boto3.resource('dynamodb', region_name = 'eu-central-1') table = dynamodb.Table('userInformations') response = table.put_item(Item) return response def lambda_handler(event, context): userAttributes = event["request"]["userAttributes"] email = userAttributes["email"] name = userAttributes["name"] Item = { "email": email, "name" : name } response = updateDynamo(Item) print(email, name) return { 'statusCode': 200, 'body': json.dumps('Hello from Lambda!') }  This is the test event i'm using: { "version": "string", "triggerSource": "string", "region": "AWSRegion", "userPoolId": "string", "userName": "string", "callerContext": { "awsSdkVersion": "string", "clientId": "string" }, "request": { "userAttributes": { "email": "testemail@gmail.com", "name": "test user" } }, "response": {} }  My table has email (all char are lowercase) as partition key.

Answer: put_item requires keyword only arguments. This means, that in your case instead of: response = table.put_item(Item)  there should be response = table.put_item(Item=Item)
Question: I'm trying to test the upload of a file and its metadata using Python and FastAPI. Here is how I defined the route for the upload: @app.post("/upload_files") async def creste_upload_files(uploaded_files: List[UploadFile], selectedModel: str = Form(...), patientId: str = Form(...), patientSex: str = Form(...), actualMedication: str = Form(...), imageDim: str = Form(...), imageFormat: str = Form(...), dateOfScan: str = Form(...)): for uploaded_dicom in uploaded_files: upload_folder = "webapp/src/data/" file_object = uploaded_dicom.file #create empty file to copy the file_object to upload_folder = open(os.path.join(upload_folder, uploaded_dicom.filename), 'wb+') shutil.copyfileobj(file_object, upload_folder) upload_folder.close() return "hello"  (I'm not using the metadata but I will later). I use unittest for the testing: class TestServer(unittest.TestCase): def setUp(self): self.client = TestClient(app) self.metadata = { "patientId": "1", "patient_age": "M", "patientSex": "59", "patient_description": "test", "actualeMedication": "test", "dateOfScan": datetime.strftime(datetime.now(), "%d/%m/%Y"), "selectedModel": "unet", "imageDim": "h", "imageFormat": "h" } def tearDown(self): pass def test_dcm_upload(self): dicom_file = pydicom.read_file("tests/data/1-001.dcm") bytes_data = dicom_file.PixelData files = {"uploaded_files": ("dicom_file", bytes_data, "multipart/form-data")} response = self.client.post( "/upload_files", json=self.metadata, files=files ) print(response.json())  But it seems that the upload doesn't work, I get the following print of the response: {'detail': [{'loc': ['body', 'selectedModel'], 'msg': 'field required', 'type': 'value_error.missing'}, {'loc': ['body', 'patientId'], 'msg': 'field required', 'type': 'value_error.missing'}, {'loc': ['body', 'patientSex'], 'msg': 'field required', 'type': 'value_error.missing'}, {'loc': ['body', 'actualMedication'], 'msg': 'field required', 'type': 'value_error.missing'}, {'loc': ['body', 'imageDim'], 'msg': 'field required', 'type': 'value_error.missing'}, {'loc': ['body', 'imageFormat'], 'msg': 'field required', 'type': 'value_error.missing'}, {'loc': ['body', 'dateOfScan'], 'msg': 'field required', 'type': 'value_error.missing'}]}  It is possible that I should upload using a Formdata instead of the body request (json=self.metadata) but I don't know how it should be done.

Answer: The answer is just to replace json=self.metadata which can be used for body parameters by data=self.metadata for a formData
Question: I used field(init= False) to disable initializing self.ref. It is then a value in post. The following code raises AttributeError: 'Data' object has no attribute 'ref' from dataclasses import dataclass, field def make_list(): return [[0] for k in range(9)] @dataclass class Data: rows: list cols: list blocks: list ref: dict = field(init=False) def __init__(self, slots=None): self.rows = make_list() self.cols = make_list() self.blocks = make_list() if slots: for i in range(9): for j in range(9): self.cols[j][i] = self.rows[i][j] = slots[i][j] def __post_init__(self): print("post-init executed") self.ref = {"rows": self.rows, "cols": self.cols, "blocks": self.blocks} test = Data() print(test)  I am using python 3.8. The code is tested in both pycharm/jupyter. (Same error) Edit: after correcting the typo: __post__init__ to __post_init__ , I am still getting the error.

Answer: Thanks to @wim and @juanpa.arrivillaga Deleting the __init__ would fix the problem and let the __post_init__ run again. (As pointed out by wim and juanpa.arrivillaga) If I write my own __init__ , why even bother writing __post_init__ , I can write all post processing all I want in there. (line order) from dataclasses import dataclass, field @dataclass class Data: rows: list cols: list = field(init=False) blocks: list = field(init=False) ref: dict = field(init=False) def __post_init__(self): print("post init\n\n") self.cols = [k*10 for k in self.rows] # code transform rows to cols self.blocks = [k*20 for k in self.rows] # code transform rows to blocks self.ref = {"rows": self.rows, "cols": self.cols, "blocks": self.blocks} test = Data([1,2,3]) print(test)  Also, I might want to reconsider rewriting it using regular class as the code stands now since using dataclass here does not provide anything more than a regular class.
Question: I'm trying to use opencv-python with GPU on windows 10. I installed opencv-contrib-python using pip and it's v4.4.0.42, I also have Cuda on my computer and in path. Anyway, here is a (simple) code that I'm trying to compile: import cvlib as cv from cvlib.object_detection import draw_bbox bbox, label, conf = cv.detect_common_objects(img,confidence=0.5,model='yolov3-worker',enable_gpu=True) output_image = draw_bbox(img, bbox, label, conf)  First, here is the line that tell me that tf is ok with cuda: 2020-08-26 5:51:55.718555: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll  but when I try to use my GPU to analyse the image, here is what happen: [ WARN:0] global C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-j8nxabm_\opencv\modules\dnn\src\dnn.cpp (1429) cv::dnn::dnn4_v20200609::Net::Impl::setUpNet DNN module was not built with CUDA backend; switching to CPU  Is there a way to solve this without install opencv using cmake? It's a mess on windows...

Answer: The problem here is that version of opencv distributed with your system (Windows in this case) was not compiled with Cuda support. Therefore, you cannot use any cuda related function with this build. If you want to have an opencv with cuda support, you will have to either compile it yourself (which may be tedious on windows) or find a prebuilt one somewhere. In case you want to go for the 1st solution, here is a link that may help you with the process: https://programming.vip/docs/compile-opencv-with-cuda-support-on-windows-10.html. Keep in mind that this will require you to install a bunch of SDK in the process.
Question: In terms of readability and performance, should I pre-allocate memory for an array using [None]*n? Is allocating an empty one [] and using .append() over and over considered wasteful?

Answer: In this simple timing test, the use of [None] * n does indeed appear to be slightly quicker, but arguably not by enough to justify adopting this approach over the more usual idioms. import time def func1(size): a = [None] * size for i in range(size): a[i] = i def func2(size): a = [] for i in range(size): a.append(i) def func3(size): a = [i for i in range(size)] size = 1000000 repeat = 100 t0 = time.time() for _ in range(repeat): func1(size) t1 = time.time() for _ in range(repeat): func2(size) t2 = time.time() for _ in range(repeat): func2(size) t3 = time.time() print(t1 - t0, t2 - t1, t3 - t2)  Results:  [None * size] and then index: 4.82 seconds append in a loop: 6.37 seconds list comprehension: 6.34 seconds  Repeating the tests with size=1000 and repeat=100000 give similar results:  [None * size] and then index: 3.16 seconds append in a loop: 4.88 seconds list comprehension: 4.84 seconds  And again with size=10 and repeat = 10000000:  [None * size] and then index: 6.09 seconds append in a loop: 7.65 seconds list comprehension: 7.66 seconds
Question: I have a question regarding asynchronous requests: How do I save response.json() to a file, on the fly? I want to make a request and save response to a .json file, without keeping it in memory.  import asyncio import aiohttp async def fetch(sem, session, url): async with sem: async with session.get(url) as response: return await response.json() # here async def fetch_all(urls, loop): sem = asyncio.Semaphore(4) async with aiohttp.ClientSession(loop=loop) as session: results = await asyncio.gather( *[fetch(sem, session, url) for url in urls] ) return results if __name__ == '__main__': urls = ( "https://public.api.openprocurement.org/api/2.5/tenders/6a0585fcfb05471796bb2b6a1d379f9b", "https://public.api.openprocurement.org/api/2.5/tenders/d1c74ec8bb9143d5b49e7ef32202f51c", "https://public.api.openprocurement.org/api/2.5/tenders/a3ec49c5b3e847fca2a1c215a2b69f8d", "https://public.api.openprocurement.org/api/2.5/tenders/52d8a15c55dd4f2ca9232f40c89bfa82", "https://public.api.openprocurement.org/api/2.5/tenders/b3af1cc6554440acbfe1d29103fe0c6a", "https://public.api.openprocurement.org/api/2.5/tenders/1d1c6560baac4a968f2c82c004a35c90", ) loop = asyncio.get_event_loop() data = loop.run_until_complete(fetch_all(urls, loop)) print(data)   For now, the script just prints JSON files, and I can save them once they're all scraped: data = loop.run_until_complete(fetch_all(urls, loop)) for i, resp in enumerate(data): with open(f"{i}.json", "w") as f: json.dump(resp, f)  But it doesn't feel right to me as it will definitely fail once I run out of memory for example. Any suggestions?  Edit Limited my post to only one question

Answer:  How do I save response.json() to a file, on the fly?  Don't use response.json() in the first place, use the streaming API instead: async def fetch(sem, session, url): async with sem, session.get(url) as response: with open("some_file_name.json", "wb") as out: async for chunk in response.content.iter_chunked(4096) out.write(chunk)
Question: Say I have a dataframe import numpy as np import pandas as pd df = pd.DataFrame(np.random.randint(10, size=(10,3)), columns=['a', 'b', 'c'])  if I now try to query it using the query method: this works: df.query('''a > 3 and b < 9''') this throws an error: df.query( ''' a > 3 and b < 9 ''' )  I tried many variations of multiline strings but the result is always the following error: ~/ven/lib/python3.6/site-packages/pandas/core/computation/eval.py in eval(expr, parser, engine, truediv, local_dict, global_dict, resolvers, level, target, inplace) 306 if multi_line and target is None: 307 raise ValueError( --> 308 "multi-line expressions are only valid in the " 309 "context of data, use DataFrame.eval" 310 ) ValueError: multi-line expressions are only valid in the context of data, use DataFrame.eval  Does anyone know how to make it work? The problem is that in reality I have a very long query to do and it would be very inconvenient having to write all in one line. I know I could use boolean indexing instead but my question is only about how to use multiline with the query method. Thank you

Answer: Use multi-line char backslash ( \ ) Ex: df = pd.DataFrame(np.random.randint(10, size=(10,3)), columns=['a', 'b', 'c']) print(df.query( ''' a > 3 and \ b < 9 ''' ))
Question: While calling the .delay() method of an imported task from a django application, the process gets stuck and the request is never completed. We also don't get any error on the console. Setting up a set_trace() with pdb results in the same thing. The following questions were reviewed which didn't help resolve the issue: Calling celery task hangs for delay and apply_async celery .delay hangs (recent, not an auth problem) Eg.: backend/settings.py CELERY_BROKER_URL = os.environ.get("CELERY_BROKER", RABBIT_URL) CELERY_RESULT_BACKEND = os.environ.get("CELERY_BROKER", RABBIT_URL)  backend/celery.py from __future__ import absolute_import, unicode_literals import os from celery import Celery # set the default Django settings module for the 'celery' program. os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'backend.settings') app = Celery('backend') app.config_from_object('django.conf:settings', namespace='CELERY') # Load task modules from all registered Django app configs. app.autodiscover_tasks() @app.task(bind=True) def debug_task(self): print('Request: {0!r}'.format(self.request))  app/tasks.py import time from celery import shared_task @shared_task def upload_file(request_id): time.sleep(request_id) return True  app/views.py from rest_framework.views import APIView from .tasks import upload_file class UploadCreateAPIView(APIView): # other methods... def post(self, request, *args, **kwargs): id = request.data.get("id", None) # business logic ... print("Going to submit task.") import pdb; pdb.set_trace() upload_file.delay(id) # <- this hangs the runserver as well as the set_trace() print("Submitted task.") 

Answer: The issue was with the setup of the celery application with Django. We need to make sure that the celery app is imported and initialized in the following file: backend\__init__.py from __future__ import absolute_import, unicode_literals # This will make sure the app is always imported when # Django starts so that shared_task will use this app. from .celery import app as celery_app __all__ = ('celery_app',)
Question: I am using the dotenv package. I had a key that I had saved in my .env file but I updated it to a new key, but my script still outputs the old key. I have the ".env" file in the root directory. I thought that by using load_dotenv() that it's taking in the new keys whatever they may be at the current state in time and saving it to be used in the script. What am I doing wrong? import os from dotenv import load_dotenv import praw load_dotenv() reddit = praw.Reddit(client_id=os.getenv('reddit_personal_use'), client_secret=os.getenv('reddit_api_key'), user_agent=os.getenv('reddit_app_name'), username=os.getenv('reddit_username'), password=os.getenv('reddit_pw')) 

Answer: I had to set override=True load_dotenv(override=True)  load_dotenv does not override existing System environment variables. To override, pass override=True to load_dotenv().
Question: I'm working on a project with many different Django apps. I want to use isort on this project but the imports from Django apps (from myapp1.mymodule import myfunction) are seen by isort as third-party imports. How can I make isort recognize them as first-party imports? I could add in the isort configuration (in the .cfg): known_first_party=myapp1,myapp2... but I'll have to maintain this list. Is there a better way?

Answer: You can use the src_paths option to specify the project folder. You do not need to maintain a known_first_party list. See the related source code: if ( _is_module(module_path) or _is_package(module_path) or _src_path_is_module(src_path, root_module_name) ): return (sections.FIRSTPARTY, f"Found in one of the configured src_paths: {src_path}.")
Question: My Folder Structure looks like this: - 95000 - 95002 - 95009 - AR_95000.pdf - AR_95002.pdf - AR_95009.pdf - BS_95000.pdf - BS_95002.pdf - BS_95009.pdf  [Note 95000, 95002, 95009 are folders]  My goal is to move files AR_95000.pdf and BS_95000.pdf to the folder named 95000, then AR_95002.pdf and BS_95002.pdf to the folder named 95002 and so on. The PDFs are reports generated by system and thus I can not control the naming.

Answer: Using pathlib this task becomes super easy: from pathlib import Path root = Path("/path/to/your/root/dir") for file in root.glob("*.pdf"): folder_name = file.stem.rpartition("_")[-1] file.rename(root / folder_name / file.name)  As you can see, one main advantage of pathlib over os/shutil (in this case) is the interface Path objects provide directly to os-like functions. This way the actual copying (rename()) is done directly as an instance method.  References:  Path.glob Path.stem str.rpartition Path.rename path concatenation
Question: I followed the video from TechWithTim step by step (https://www.youtube.com/watch?v=cnPlKLEGR7E) but I am still getting an error when I try to open the sheet. The code works fine until sheet = client.open("GuildTaxes").sheet1 line. Here is my code. import gspread from oauth2client.service_account import ServiceAccountCredentials scope = ["https://spreadsheets.google.com/feeds","https://www.googleapis.com/auth/sprea...", "https://www.googleapis.com/auth/drive...","https://www.googleapis.com/auth/drive"] creds = ServiceAccountCredentials.from_json_keyfile_name("GuildTaxes-9ba4508be840.json", scope) client = gspread.authorize(creds) sheet = client.open("GuildTaxes").sheet1 data = sheet.get_all_records() print(data) 

Answer: I found the answer! After 2 hours, the scope in TechWithTim's video doesn't work for me, so if you stumble upon the same issue try using this one scope = [ 'https://www.googleapis.com/auth/spreadsheets', 'https://www.googleapis.com/auth/drive' ]  It is the default scope.