eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am able to make histogram in python but I am unable to add density curve , I see many code which are using different ways to add density curve on histogram but I am not sure how to get on my code I have added density = true but not able to get density curve on histogram df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD')) X=df['A'] hist, bins = np.histogram(X, bins=10,density=True) width = 0.7 * (bins[1] - bins[0]) center = (bins[:-1] + bins[1:]) / 2 plt.bar(center, hist, align='center', width=width) plt.show() 

Answer: distplot has been removed: removed in a future version of seaborn. Therefore, alternatives are to use histplot and displot. sns.histplot import pandas as pd import matplotlib.pyplot as plt import seaborn as sns import numpy as np df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD')) X = df['A'] sns.histplot(X, kde=True, bins=20) plt.show()   sns.displot import pandas as pd import matplotlib.pyplot as plt import seaborn as sns import numpy as np df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD')) X = df['A'] sns.displot(X, kde=True, bins=20) plt.show()    distplot has been removed Here is an approach using distplot method of seaborn. Also, mentioned in the comments: import pandas as pd import matplotlib.pyplot as plt import seaborn as sns import numpy as np df = pd.DataFrame(np.random.randn(100, 4), columns=list('ABCD')) X = df['A'] sns.distplot(X, kde=True, bins=20, hist=True) plt.show()
Question: From the pydantic docs I understand this: import pydantic class User(pydantic.BaseModel): id: int name: str class Student(pydantic.BaseModel): semester: int # this works as expected class Student_User(User, Student): building: str print(Student_User.__fields__.keys()) #> dict_keys(['semester', 'id', 'name', 'building'])  However, when I want to create a similar object dynamically (following the section dynamic-model-creation): # this results in a TypeError pydantic.create_model("Student_User2", __base__=(User, Student))  I get: TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases  Question: How to dynamically create a class like Student_User

Answer: As of pydantic==1.9.2, Student_User2 = pydantic.create_model("Student_User2", __base__=(User, Student), building=(str, ...))  runs successfully and print(Student_User2.__fields__.keys())  returns dict_keys(['semester', 'id', 'name', 'building'])
Question: I'm training an image classification model with PyTorch Lightning and running on a machine with more than one GPU, so I use the recommended distributed backend for best performance ddp (DataDistributedParallel). This naturally splits up the dataset, so each GPU will only ever see one part of the data. However, for validation, I would like to compute metrics like accuracy on the entire validation set and not just on a part. How would I do that? I found some hints in the official documentation, but they do not work as expected or are confusing to me. What's happening is that validation_epoch_end is called num_gpus times with 1/num_gpus of the validation data each. I would like to aggregate all results and only run the validation_epoch_end once. In this section they state that when using dp/ddp2 you can add an additional function called like this def validation_step(self, batch, batch_idx): loss, x, y, y_hat = self.step(batch) return {"val_loss": loss, 'y': y, 'y_hat': y_hat} def validation_step_end(self, self, *args, **kwargs): # do something here, I'm not sure what, # as it gets called in ddp directly after validation_step with the exact same values return args[0]  However, the results are not being aggregated and validation_epoch_end is still called num_gpu times. Is this kind of behavior not available for ddp? Is there some other way how achieve this aggregation behavior?

Answer: training_epoch_end() and validation_epoch_end() receive data that is aggregated from all training / validation batches of the particular process. They simply receive a list of what you returned in each training or validation step. When using the DDP backend, there's a separate process running for every GPU. There's no simple way to access the data that another process is processing, but there's a mechanism for synchronizing a particular tensor between the processes. The easiest approach for computing a metric on the entire validation set is to calculate the metric in pieces and then synchronize the resulting tensor, for example by taking the average. self.log() calls will automatically synchronize the value between GPUs when you use sync_dist=True. How the value is synchronized is determined by the reduce_fx argument, which by default is torch.mean. If you're happy with averaging the metric over batches too, you don't need to override training_epoch_end() or validation_epoch_end() — self.log() will do the averaging for you. If the metric cannot be calculated separately for each GPU and then averaged, it can get a bit more challenging. It's possible to update some state variables at each step, and then synchronize the state variables at the end of an epoch and calculate the metric. The recommended way is to create a class that derives from the Metric class from the TorchMetrics project. Add the state variables in the constructor using add_state() and override the update() and compute() methods. The API will take care of synchronizing the state variables between the GPU processes. There's already an accuracy metric in TorchMetrics and the source code is a good example of how to use the API.
Question: Im trying to create connection between flask socketio and react native socketio, I have already prepared client side with react native socketio BUT I have run into problem with importing flask_socketio in rpi. Im trying to use simplest implementation possible, this is my code:   from flask import Flask from flask_socketio import SocketIO, emit app = Flask(__name__) @app.route('/', methods=['GET']) def hello_world(): return "Hello World" if __name__ == '__main__': app.run(host='0.0.0.0', port=5005)    Without line 2 it works perfectly but I need to use flask_socketio. here is photo too(first run is without importing flask_socketio, than I tried to import it and it wont work. I have tried reinstalling flask_socketio twice, rebooting but nothing works:/:

Answer: Problem was that I was installing it with sudo python pip install flask_socketio, BUT I had to use python3, so right installation is python3 -m pip install flask_socketio
Question: I'm trying to dump my entire DB to a json. When I run python manage.py dumpdata > data.json I get an error: (env) PS C:\dev\watch_something> python manage.py dumpdata > data.json CommandError: Unable to serialize database: 'charmap' codec can't encode character '\u0130' in position 1: character maps to <undefined> Exception ignored in: <generator object cursor_iter at 0x0460C140> Traceback (most recent call last): File "C:\dev\watch_something\env\lib\site-packages\django\db\models\sql\compiler.py", line 1602, in cursor_iter cursor.close() sqlite3.ProgrammingError: Cannot operate on a closed database.  It's because one of the characters in my DB is a sepcial character. How can I dump the DB correctly? FYI, all other DB functionalities work fine

Answer: One solution is to use ./manage.py dumpdata -o data.json instead of ./manage.py dumpdata > data.json. Another solution is to use Python's UTF-8 mode, run: python -Xutf8 ./manage.py dumpdata > data.json
Question: I've never seen an error like this and don't know where to start. I installed opencv with conda install opencv  and am running Ubuntu Linux 18.04 using a conda environment named fpn. How should I even approach debugging this? Traceback (most recent call last): File "test.py", line 5, in <module> import cv2 ImportError: /home/s/miniconda3/envs/fpn/lib/python3.7/site-packages/../../././libgnutls.so.30: symbol mpn_add_1 version HOGWEED_4 not defined in file libhogweed.so.4 with link time reference 

Answer: There seems to be a problem with the recent releases of opencv packages for Conda. I have tested all the 4.x releases and found that the problem occurs starting from 4.3. Unless you really depend on >=4.3, forcing a version prior to 4.3 solves the problem, name: test channels: - anaconda - conda-forge dependencies: - python>=3.8 - opencv<4.3  in my cases this installed 4.2.0. Importing cv2 in Python works fine then. Note that using conda update didn't work for me and I still got the error, but I had to first remove the environment and then re-create it. I think this behavior indicates that the error is rooted in some dependency of opencv, which is not properly down-graded when conda update is used.
Question: I am trying to run survival analysis in python (pycharm) in linux, here is a part of the code import numpy as np import matplotlib.pyplot as plt #matplotlib inline import pandas as pd from sklearn.impute import SimpleImputer from sklearn.pipeline import make_pipeline from sklearn.model_selection import train_test_split from sksurv.datasets import load_flchain from sksurv.linear_model import CoxPHSurvivalAnalysis  I get the error "ModuleNotFoundError: No module named 'sksurv'", I tried everything, but nothing works.

Answer: The required dependencies for scikit-survival,  cvxpy  cvxopt  joblib  numexpr  numpy 1.12 or later  osqp  pandas 0.21 or later  scikit-learn 0.22  scipy 1.0 or later   ...will be automatically installed by pip when you run: pip install scikit-survival  However, one module in particular, osqp, has CMake as one of its dependencies. If you don't have CMake installed, pip install scikit-survival will throw an error and the installation will fail. You can download CMake for your OS at cmake.org/download After CMake has installed, you should be able to successfully run pip install scikit-survival  Notes:  GCC needs to be installed also scikit-survival works with Python 3.5 or higher More information is available in the docs
Question: I want use progress bars in my python code. I know there are many libraries for that but I want to use the progress bars used by pip [the package manager]. Please tell if there is a way to do this.

Answer: The progress package available on pypi is used by pip. It can be imported by including the following line in your python file: from pip._vendor import progress  Usage is available on https://pypi.org/project/progress/
Question: Word of notice: This is my first approach with asyncio, so I might have done something really stupid. Scenario is as follows: I need to "http-ping" a humongous list of urls to check if they respond 200 or any other value. I get timeouts for each and every request, though tools like gobuster report 200,403, etc. My code is sth similar to this: import asyncio,aiohttp import datetime #------------------------------------------------------------------------------------- async def get_data_coroutine(session,url,follow_redirects,timeout_seconds,retries): #print('#DEBUG '+datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')+' '+url) try: async with session.get(url,allow_redirects=False,timeout=timeout_seconds) as response: status = response.status #res = await response.text() if( status==404): pass elif(300<=status and status<400): location = str(response).split("Location': \'")[1].split("\'")[0] print('#HIT '+datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')+' '+str(status)+' '+url+' ---> '+location) if(follow_redirects==True): return await get_data_coroutine(session,location,follow_redirects,timeout_seconds,retries) else: print('#HIT '+datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')+' '+str(status)+' '+url) return None except asyncio.exceptions.TimeoutError as e: print('#ERROR '+datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')+' '+' '+' '+url+' TIMEOUT '+str(e)) return None #--------------------------------------------------------------------------- async def main(loop): base_url = 'http://192.168.59.37' extensions = ['','.html','php'] fd = open('/usr/share/wordlists/dirb/common.txt','r') words_without_suffix = [x.strip() for x in fd.readlines()]#[-5:] #DEBUG! words_with_suffix = [base_url+'/'+x+y for x in words_without_suffix for y in extensions] follow = True total_timeout = aiohttp.ClientTimeout(total=60*60*24) timeout_seconds = 10 retries = 1 async with aiohttp.ClientSession(loop=loop,timeout=total_timeout) as session: tasks = [get_data_coroutine(session,url,follow,timeout_seconds,retries) for url in words_with_suffix] await asyncio.gather(*tasks) print('DONE') #--------------------------------------------------------------------------- if(__name__=='__main__'): loop = asyncio.get_event_loop() result = loop.run_until_complete(main(loop))  Did I do something really wrong? Any word of advice? Thank you SO much!

Answer: Actually, I ended up finding an open issue in aio-libs/aiohttp: https://github.com/aio-libs/aiohttp/issues/3203 This way, they suggest a workaround that achieves my needs: session_timeout = aiohttp.ClientTimeout(total=None,sock_connect=timeout_seconds,sock_read=timeout_seconds) async with aiohttp.ClientSession(timeout=session_timeout) as session: async with session.get(url,allow_redirects=False,timeout=1) as response: ...
Question: I have got a simple Lambda code to read the csv file from S3 Bucket. All is working fine however I tried to get the csv data to pandas data frame and the error comes up string indices must be integers My code is bog-standard but I just need to use the csv as a data frame for further manipulation. The hashed line is the source of the error. I can print data with no problems so the bucket and file details are configured properly. updated code import json import pandas as pd import numpy as np import requests import glob import time import os from datetime import datetime from csv import reader import boto3 import traceback import io s3_client = boto3.client('s3') def lambda_handler(event, context): try: bucket_name = event["Records"][0]["s3"]["bucket"]["name"] s3_file_name = event["Records"][0]["s3"]["object"]["key"] resp = s3_client.get_object(Bucket=bucket_name, Key=s3_file_name) data = resp['Body'].read().decode('utf-8') df=pd.DataFrame( list(reader(data))) print (df.head()) except Exception as err: print(err) # TODO implement return { 'statusCode': 200, 'body': json.dumps('Hello fr2om Lambda!') } traceback.print_exc() 

Answer: I believe that your problem is likely tied to this line - df=pd.DataFrame( list(reader(data))) in your function. The answer below should allow you to read the csv file into the pandas dataframe for processes. import boto3 import pandas as pd from io import BytesIO s3_client = boto3.client('s3') def lambda_handler(event, context): try: bucket_name = event["Records"][0]["s3"]["bucket"]["name"] s3_file_name = event["Records"][0]["s3"]["object"]["key"] resp = s3_client.get_object(Bucket=bucket_name, Key=s3_file_name) ########################################### # one of these methods should work for you. # Method 1 # df_s3_data = pd.read_csv(resp['Body'], sep=',') # # Method 2 # df_s3_data = pd.read_csv(BytesIO(resp['Body'].read().decode('utf-8'))) ########################################### print(df_s3_data.head()) except Exception as err: print(err)
Question: I want to add an auth_required decorator to my endpoints. (Please consider that this question is about decorators, not middleware) So a simple decorator looks like this: def auth_required(func): def wrapper(*args, **kwargs): if user_ctx.get() is None: raise HTTPException(...) return func(*args, **kwargs) return wrapper  So there are 2 usages: @auth_required @router.post(...)  or @router.post(...) @auth_required  The first way doesn't work because router.post creates a router that saved into self.routes of APIRouter object. The second way doesn't work because it fails to verify pydantic object. For any request model, it says missing args, missing kwargs. So my question is - how can I add any decorators to FastAPI endpoints? Should I get into router.routes and modify the existing endpoint? Or use some functools.wraps like functions?

Answer:  How can I add any decorators to FastAPI endpoints?  As you said, you need to use @functools.wraps(...)--(PyDoc) decorator as, from functools import wraps from fastapi import FastAPI from pydantic import BaseModel class SampleModel(BaseModel): name: str age: int app = FastAPI() def auth_required(func): @wraps(func) async def wrapper(*args, **kwargs): return await func(*args, **kwargs) return wrapper @app.post("/") @auth_required # Custom decorator async def root(payload: SampleModel): return {"message": "Hello World", "payload": payload} The main caveat of this method is that you can't access the request object in the wrapper and I assume it is your primary intention. If you need to access the request, you must add the argument to the router function as, from fastapi import Request @app.post("/") @auth_required # Custom decorator async def root(request: Request, payload: SampleModel): return {"message": "Hello World", "payload": payload}  I am not sure what's wrong with the FastAPI middleware, after all, the @app.middleware(...) is also a decorator.
Question: In the project I am currently working on, I am not allowed to use an ORM so I made my own It works great but I am having problems with Celery and it's concurrency. For a while, I had it set to 1 (using --concurrency=1) but I'm adding new tasks which take more time to process than they need to be run with celery beat, which causes a huge backlog of tasks. When I set celery's concurrency to > 1, here's what happens (pastebin because it's big): https://pastebin.com/M4HZXTDC Any idea on how I could implement some kind of lock/wait on the other processes so that the different workers don't cross each other? Edit: Here is where I setup my PyMySQL instance and how the open and close are handled

Answer: PyMSQL does not allow threads to share the same connection (the module can be shared, but threads cannot share a connection). Your Model class is reusing the same connection everywhere. So, when different workers call on the models to do queries, they are using the same connection object, causing conflicts. Make sure your connection objects are thread-local. Instead of having a db class attribute, consider a method that will retrieve a thread-local connection object, instead of reusing one potentially created in a different thread. For instance, create your connection in the task. Right now, you're using a global connection everywhere for every model. # Connect to the database connection = pymysql.connect(**database_config) class Model(object): """ Base Model class, all other Models will inherit from this """ db = connection  To avoid this you can create the DB in the __init__ method instead... class Model(object): """ Base Model class, all other Models will inherit from this """ def __init__(self, *args, **kwargs): self.db = pymysql.connect(**database_config)  However, this may not be efficient/practical because every instance of the db object will create a session. To improve upon this, you could use an approach using threading.local to keep connections local to threads.  class Model(object): """ Base Model class, all other Models will inherit from this """ _conn = threading.local() @property def db(self): if not hasattr(self._conn, 'db'): self._conn.db = pymysql.connect(**database_config) return self._conn.db  Note, a thread-local solution works assuming you're using a threading concurrency model. Note also that celery uses multiple processes (prefork) by default. This may or may not be a problem. If it is a problem, you may be able to work around it if you change the workers to use eventlet instead.
Question: Here's my situation. I have some jupyter notebooks inside some folder and I would like to share some code between those notebooks trough a library I made. The folder structure is the following: 1.FirstFolder/ notebookA.ipynb 2.SecondFolder/ notebookB.ipynb mylib/ __init__.py otherfiles.py  I tried putting the following code at the beginning of the notebook: # to use modules in parent folder import sys import os from pathlib import Path libpath = os.path.join(Path.cwd().parent,'mylib') print(f"custom library functions are in the module:\n\t{libpath}") sys.path.append(libpath) import mylib  The print outputs the correct path of the module and then a ModuleNotFoundError comes up and the program crashes: ---> 10 import mylib 11 from mylib import * ModuleNotFoundError: No module named 'mylib'  Looking up on SO I found that that should have been the way to import a module from a non-default folder. Where is the error? EDIT: after FinleyGibson's answer I tried sys.path.append(Path.cwd().parent) and restarted the kernel but I still have the same problem. EDIT2: I tried this and it worked, but I still would like to know why the previous approaches haven't worked. import sys import os from pathlib import Path tmp = Path.cwd() os.chdir(Path.cwd().parent) sys.path.append(Path.cwd()) import mylib from mylib.dataloading import * os.chdir(tmp) 

Answer: You have added the contents of os.path.join(Path.cwd().parent,'mylib') to your path, this means python will look inside this dir for the module you are importing. mylib is not located in this dir, but rather the parent dir. Also Path.cwd().parent returns a pathlib.PosixPath object. Convert this to a string to use it with import (or, just use sys.path.append('../'): try: import sys import os from pathlib import Path sys.path.append(str(Path.cwd().parent)) import mylib  doing this allows me to import a variable X = 'import success' located in otherfiles.py like so: ans = mylib.otherfiles.X print(ans) >>> 'import success'
Question: I got the following mail from SentGrid,  We are emailing to inform you of an upcoming requirement to update your authentication method with Twilio SendGrid to API keys exclusively by December 9th, 2020 in order to ensure uninterrupted service and improve the security of your account. Our records show that you have used basic authentication with username and password for one or more of your API requests with 1 users of your SendGrid account in the last 180 days.   Why API keys?   This is an effort to enhance security for all of our users. Using your account username and password for authentication is less secure than using an API Key. Unlike your username and password, API Keys are uniquely generated and can be set to limit the access and specify permissions for a given request.   What action is required?   Follow these steps to identify and replace your authentication method to API Keys and then implement Two-Factor Authentication (2FA) for enhanced security. What happens if no action is taken? On December 9th, 2020 we will no longer accept basic authentication with username and password, and we will be requiring 2FA to login to your account. If you attempt to authenticate your API requests or SMTP configuration with username and password for any of your users after that date, your requests will be rejected. We’d like to thank you in advance for your prompt attention to these requirements. If you’d like to learn more about how you can enhance the security of your account, view this post. If you have any questions or need assistance, please visit our documentation or reach out to our Support team. Thank you, The Twilio SendGrid Team  Presently I am sending mails to sendgrid by using following credentials, EMAIL_HOST = 'smtp.sendgrid.net' EMAIL_USE_TLS = False EMAIL_PORT = 587 EMAIL_HOST_USER = 'xx@gmail.com' EMAIL_HOST_PASSWORD = 'xxx''' Is this change affect me? 

Answer: Yes, once they force two factor authentication (2FA), your application will not be able to do basic authentication by just using username/email & password. So, you need to start using API keys. Migration is simple:  Login to sendgrid account Goto https://app.sendgrid.com/settings/api_keys "Generate API Key" - generate a new API key and copy paste to be used later Code changes:  EMAIL_HOST_USER = 'apikey' (username should be this only) EMAIL_HOST_PASSWORD = 'YOUR_API_KEY'   Test it  If the changes work, you are good to go and have migrated from basic authentication to API keys.