eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am using Python 3.8.1 and mypy 0.782. I don't understand why mypy complains about the following code: from typing import Union, List, Dict Mytype = Union[Dict[str, str], Dict[str, List[str]]] s: Mytype = {"x": "y", "a": ["b"]}  Mypy gives the following error on line 3: Incompatible types in assignment (expression has type "Dict[str, Sequence[str]]", variable has type "Union[Dict[str, str], Dict[str, List[str]]]")  If I change the last line to s: Mytype = {"a": ["b"]} mypy doesn't complain. However, when adding yet one more line s["a"].append("c") leads to an error: error: Item "str" of "Union[str, List[str]]" has no attribute "append"  How can the above mentioned be explained? How should I type a dict that has strings as keys and the values can be either strings or lists of strings? Found this: https://github.com/python/mypy/issues/2984#issuecomment-285716826 but still not completely sure why the above mentioned happens and how should I fix it. EDIT: Although it's still not clear why the suggested modification Mytype = Dict[str, Union[str, List[str]]] does not resolve the error with s['a'].append('c') I think the TypeDict approach suggested in the comments as well as in https://stackoverflow.com/a/62862029/692695 is the way to go, so marking that approach as a solution. See similar question at: Indicating multiple value in a Dict[] for type hints, suggested in the comments by Georgy.

Answer: Because s: Mytype cannot have type Dict[str, str] and type Dict[str, List[str]] at the same time. You could do what you want like this: Mytype = Dict[str, Union[str, List[str]]]  But maybe problems, because Dict is invariant  Also you could use TypedDict, but only a fixed set of string keys is expected: from typing import List, TypedDict MyType = TypedDict('MyType', {'x': str, 'a': List[str]}) s: MyType = {"x": "y", "a": ["b"]} s['a'].append('c')  NOTE:  Unless you are on Python 3.8 or newer (where TypedDict is available in standard library typing module) you need to install typing_extensions using pip to use TypedDict   And, of course, you can use Any: Mytype = Dict[str, Any]
Question: I have a function which returns a function. I would like to find a proper type annotation. However, the returned function has *args and *kwargs. How is that annotated within Callable[[Parameters???], ReturnType]? Example: from typing import Callable import io import pandas as pd def get_conversion_function(file_type: str) -> Callable[[io.BytesIO, TODO], pd.DataFrame]: def to_csv(bytes_, *args, **kwargs): return pd.read_csv(bytes_, **kwargs) if file_type == "csv": return to_csv 

Answer: As I know, python's typing does not allow do that straightforwardly as stated in the docs of typing.Callable:  There is no syntax to indicate optional or keyword arguments; such function types are rarely used as callback types. Callable[..., ReturnType] (literal ellipsis) can be used to type hint a callable taking any number of arguments and returning ReturnType.  But you could use mypy extensions like this: from typing import Callable from mypy_extensions import Arg, VarArg, KwArg def foo(a: str, *args: int, **kwargs: float) -> str: return 'Hello, {}'.format(a) def bar() -> Callable[[Arg(str, 'a'), VarArg(int), KwArg(float)], str]: return foo
Question: I'd like to add a trace to all facets of a plotly plot. For example, I'd like to add a reference line to each daily facet of a scatterplot of the "tips" dataset showing a 15% tip. However, my attempt below only adds the line to the first facet. import plotly.express as px import plotly.graph_objects as go import numpy as np df = px.data.tips() ref_line_slope = 0.15 # 15% tip for reference ref_line_x_range = np.array([df.total_bill.min(), df.total_bill.max()]) fig = px.scatter(df, x="total_bill", y="tip",facet_col="day", trendline='ols') fig = fig.add_trace(go.Scatter(x=reference_line_x_range,y=ref_line_slope*reference_line_x_range,name='15%')) fig.show()  

Answer: According to an example from plotly you can pass 'all' as the row and col arguments and even skip empty subplots: fig.add_trace(go.Scatter(...), row='all', col='all', exclude_empty_subplots=True)
Question: I have a simple API function as below, from fastapi import FastAPI app = FastAPI() @app.get("/") async def read_root(): return {"Hello": "World"}  I am starting the server using uvicorn command as, uvicorn main:app  Since we are not calling any python file directly, it is not possible to call uvicorn command from Pycharm. So, How can I run the fast-api server using Pycharm?

Answer: Method-1: Run FastAPI by calling uvicorn.run(...) In this case, your minimal code will be as follows, # main.py import uvicorn from fastapi import FastAPI app = FastAPI() @app.get("/") async def read_root(): return {"Hello": "World"} if __name__ == "__main__": uvicorn.run(app, host="0.0.0.0", port=8000) Normally, you'll start the server by running the following command, python main.py  Pycharm Setup For this setup, and now, you can set the script path in Pycharm's config  Notes  Script Path: path to the FastAPI script Python Interpreter: Choose your interpreter/virtual environment Working Directory: Your FastAPI project root   Method-2: Run FastAPI by calling uvicorn command In this case, your minimal code will be as follows, # main.py from fastapi import FastAPI app = FastAPI() @app.get("/") async def read_root(): return {"Hello": "World"} Normally, you'll start the server by running the following command, uvicorn main:app --reload  Pycharm Setup For this setup, and now, you can set the script path in Pycharm's config  Notes  Module name: set to uvicorn [Optional] Script: Path to uvicorn binary. You will get the path by executing the command, which uvicorn , inside your environment. (See this image) Parameters: The actual parameters of uvicorn command Python Interpreter: Choose your interpreter/virtual environment Working Directory: Your FastAPI project root
Question: In terminal  windows 10 using cuda 10.1 python 3.7.7 GPU GeForce GTX 1050 4GB  >>> import tensorflow as tf 2020-07-08 17:10:50.005569: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll >>> tf.config.list_physical_devices('GPU') 2020-07-08 17:10:55.657489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:01:00.0 name: GeForce GTX 1050 computeCapability: 6.1 coreClock: 1.493GHz coreCount: 5 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 104.43GiB/s 2020-07-08 17:10:55.701387: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]  In jupyter notebook in [1]: import tensorflow as tf tf.config.list_physical_devices('GPU') out[1]: [] in [1]: tf.__version__ out[1]: '2.2.0' 

Answer: It needs to make new kernel for this env and select kernel form jupyter notebook $ conda activate env_name $ pip install ipykernel --user $ python -m ipykernel install --user --name env_name --display-name env_name
Question: I have DF that has multiple columns. Two of the columns are list of the same len.( col2 and col3 are list. the len of the list is the same). My goal is to list each element on it's own row. I can use the df.explode(). but it only accepts one column. However, I want the pair of the two columns to be 'exploded'. If I do df.explode('col2') and then df.explode('col3'), it results it 9 rows instead of 3. Original DF col0 col1 col2 col3 1 aa [1,2,3] [1.1,2.2,3.3] 2 bb [4,5,6] [4.4,5.5,6.6] 3 cc [7,8,9] [7.7,8.8,9.9] 3 cc [7,8,9] [7.7,8.8,9.9]  End DataFrame id col1 col2 col3 1 aa 1 1.1 1 aa 2 2.2 1 aa 3 3.3 2 bb 4 4.4 2 bb 5 5.5 2 bb 6 6.6 3 cc ... ...  Update None of the column have unique values, so can't be used as index.

Answer: You could set col1 as index and apply pd.Series.explode across the columns: df.set_index('col1').apply(pd.Series.explode).reset_index()  Or: df.apply(pd.Series.explode) col1 col2 col3 0 aa 1 1.1 1 aa 2 2.2 2 aa 3 3.3 3 bb 4 4.4 4 bb 5 5.5 5 bb 6 6.6 6 cc 7 7.7 7 cc 8 8.8 8 cc 9 9.9 9 cc 7 7.7 10 cc 8 8.8 11 cc 9 9.9
Question: Problem Hi all, As my title suggested it, I would like to get access to the notebook name (in jupyter-lab) as a variable. So I could reuse it in the notebook itself (for example to name some figure files generated in the notebook). I saw that a similar issue was opened years ago [see here]. However I didnt find a satisfactory answer. I like the simplicity of the answer suggested by @bill: import ipyparams  currentNotebook = ipyparams.notebook_name  However, it doesn't work for me. I got this warning the first time I execute the first cell: import ipyparams Javascript Error: Jupyter is not defined  currentNotebook = ipyparams.notebook_name  currentNotebook ''  Then if I rerun the cell again, I don't have the warning message anymore but the variable currentNotebook is still empty. (I run the cell sequentially, I didn't do a 'Run All Cells'). Configuration details My Jupyter version is jupyter notebook --version 6.0.3] jupyter-lab --version 2.1.1  I am using my notebook mostly for python code.  Edit 27/01/2021 @juan solution [here], using ipynbname is working for jupyter-notebook : 6.1.6 jupyter lab : 2.2.6  but this solution is still not working for jupyter lab : 3.0.1  Edit 28/01/2021 ipynbname is now working for jupyter 3 More details about it [here]

Answer: As an alternative you can use the following library: ipynbname #! pip install ipynbname import ipynbname nb_fname = ipynbname.name() nb_path = ipynbname.path()  This worked for me and the solution is quite straightforward.
Question: What is the difference between %time and %%time in a Jupyter Notebook Cell?

Answer: %time measures execution time of the next line. %%time measures execution time of the whole cell. For instance: %time a = 1 time.sleep(5)   CPU times: user 8 µs, sys: 0 ns, total: 8 µs Wall time: 16.9 µs  %%time a = 1 time.sleep(5)   CPU times: user 1.13 ms, sys: 2.11 ms, total: 3.24 ms Wall time: 5 s
Question: Getting this memory error. But the book/link I am following doesn't get this error. A part of Code: from sklearn.linear_model import SGDClassifier sgd_clf = SGDClassifier() sgd_clf.fit(x_train, y_train)  Error: MemoryError: Unable to allocate 359. MiB for an array with shape (60000, 784) and data type float64 I also get this error when I try to scale the data using StandardScaler's fit_transfrom But works fine in both if I decrease the size of training set (something like : x_train[:1000] ,y_train[:1000]) Link for the code in the book here. The error I get is in Line 60 and 63 (In [60] and In [63]) The book : Aurélien Géron - Hands-On Machine Learning with Scikit-Learn Keras and Tensorflow 2nd Ed (Page : 149 / 1130) So here's my question : Does this has anything to do with my ram? and what does "Unable to allocate 359" mean? is it the memory size ? Just in case my specs : CPU - ryzen 2400g , ram - 8gb (3.1gb is free when using jupyter notebook)

Answer: Upgrading python-64 bit seems to have solved all the "Memory Error" problem.
Question: I am getting an error(Blocked access to the file) in HTML to pdf conversion using pdfkit library while using a local image in my HTML file. How can I use local images in my HTML file?

Answer: I faced the same problem. I solved it by adding "enable-local-file-access" option to pdfkit.from_file(). options = { "enable-local-file-access": None } pdfkit.from_file(html_file_name, pdf_file_name, options=options)
Question: I am trying to convert a new column in a dataframe through a function based on the values in the date column, but get an error indicating "Timestamp object has no attribute dt." However, if I run this outside of a function, the dt attribute works fine. Any guidance would be appreciated. This code runs with no issues: sample = {'Date': ['2015-07-02 11:47:00', '2015-08-02 11:30:00']} dftest = pd.DataFrame.from_dict(sample) dftest['Date'] = pd.to_datetime(dftest['Date']) display(dftest.info()) dftest['year'] = dftest['Date'].dt.year dftest['month'] = dftest['Date'].dt.month  This code gives me the error message: sample = {'Date': ['2015-07-02 11:47:00', '2015-08-02 11:30:00']} dftest = pd.DataFrame.from_dict(sample) dftest['Date'] = pd.to_datetime(dftest['Date']) def CALLYMD(dftest): if dftest['Date'].dt.month>9: return str(dftest['Date'].dt.year) + '1231' elif dftest['Date'].dt.month>6: return str(dftest['Date'].dt.year) + '0930' elif dftest['Date'].dt.month>3: return str(dftest['Date'].dt.year) + '0630' else: return str(dftest['Date'].dt.year) + '0331' dftest['CALLYMD'] = dftest.apply(CALLYMD, axis=1)  Lastly, I'm open to any suggestions on how to make this code better as I'm still learning.

Answer: I'm guessing you should remove .dt in the second case. When you do apply it's applying to each element, .dt is needed when it's a group of data, if it's only one element you don't need .dt otherwise it will raise {AttributeError: 'Timestamp' object has no attribute 'dt'} reference: https://stackoverflow.com/a/48967889/13720936
Question: In python3.8, what's the difference between ImportError and ModuleNotFoundError? I'm just wondering what the difference is and why they matter.

Answer: According to the python docs:  The ImportError is raised when an import statement has trouble successfully importing the specified module. Typically, such a problem is due to an invalid or incorrect path, which will raise a ModuleNotFoundError in Python 3.6 and newer versions.
Question: I am writing a code to change the color of hair in the facial picture of a person. Doing this I made a model and was able to get a mask of the parts of the hair. But now I am stuck at a problem how to change the color of it. Below is the output mask and input image passed.   Can you suggest me the method that could be used to change the color of the hair into different colors?

Answer: Since they both have the same shape, you can mask the image of the face using mask image. We first need to perform binary thresholding on it, so it can be used as a b&w mask. Then we can perform boolean indexing based on whether a value is 0 or 255, and assign a new color, such as green? import cv2 mask = cv2.imread('eBB2Q.jpg') face = cv2.imread('luraB.jpg') _, mask = cv2.threshold(mask, thresh=180, maxval=255, type=cv2.THRESH_BINARY) # copy where we'll assign the new values green_hair = np.copy(face) # boolean indexing and assignment based on mask green_hair[(mask==255).all(-1)] = [0,255,0] fig, ax = plt.subplots(1,2,figsize=(12,6)) ax[0].imshow(cv2.cvtColor(face, cv2.COLOR_BGR2RGB)) ax[1].imshow(cv2.cvtColor(green_hair, cv2.COLOR_BGR2RGB))   Now we can combine the new image with the original using cv2.addWeighted, which will return the weighted sum of both images, hence we'll only see a difference on the masked region: green_hair_w = cv2.addWeighted(green_hair, 0.3, face, 0.7, 0, green_hair) fig, ax = plt.subplots(1,2,figsize=(12,6)) ax[0].imshow(cv2.cvtColor(face, cv2.COLOR_BGR2RGB)) ax[1].imshow(cv2.cvtColor(green_hair_w, cv2.COLOR_BGR2RGB))   Note that you can set the weights in the weighted sum via the alpha and beta parameters, depending on how much you want the new colour to predominate. Note that, as mentioned earlier the new image will be obtained from the weighted sum dst = src1*alpha + src2*beta + gamma. Let's try with another colour and setting the weights as a convex combination with alpha values ranging from say 0.5 and 0.9: green_hair = np.copy(face) # boolean indexing and assignment based on mask green_hair[(mask==255).all(-1)] = [0,0,255] fig, axes = plt.subplots(2,2,figsize=(8,8)) for ax, alpha in zip(axes.flatten(), np.linspace(.6, .95, 4)): green_hair_w = cv2.addWeighted(green_hair, 1-alpha, face, alpha, 0, green_hair_w) ax.imshow(cv2.cvtColor(green_hair_w, cv2.COLOR_BGR2RGB)) ax.axis('off')
Question: I'm trying to convert a few float columns to int in a DF but I'm getting above error. I've tried both to convert it as well as to fillna to 0(which I prefer not to do, as in my dataset the NA is required). What am I doing wrong? I've tried both: orginalData[NumericColumns] = orginalData[NumericColumns].astype('Int64') #orginalData[NumericColumns] = orginalData[NumericColumns].fillna(0).astype('Int64')  but it keeps resulting in the same error TypeError: cannot safely cast non-equivalent float64 to int64  What can I do to convert the columns?

Answer: import numpy as np orginalData[NumericColumns] = orginalData[NumericColumns].fillna(0).astype(np.int64, errors='ignore')  For NaNs you need to replace the NaNs with 0, then do the type casting
Question: I am trying to use the openCV VideoWriter class to generate a video from numpy arrays. I am using the following code: import numpy as np import cv2 size = 720*16//9, 720 duration = 2 fps = 25 out = cv2.VideoWriter('output.avi', cv2.VideoWriter_fourcc(*'X264'), fps, size) for _ in range(fps * duration): data = np.random.randint(0, 256, size, dtype='uint8') out.write(data) out.release()  The codec seems to be installed as ffmpeg can do conversions to the x264 codec and libx264 is installed. The code runs without warnings, however the videos generated seem to contain no data since I always get the following message when trying to read them with mpv:  [ffmpeg/demuxer] avi: Could not find codec parameters for stream 0 (Video: h264 (X264 / 0x34363258), none, 1280x720): unspecified pixel format  What could be the cause of this issue?

Answer: The first issue is that you are trying to create a video using black and white frames while VideoWriter assumes color by default. VideoWriter has a 5th boolean parameter where you can pass False to specify that the video is to be black and white. The second issue is that the dimensions that cv2 expects are the opposite of numpy. Thus, size should be (size[1], size[0]). A possible further problem is the codec being used. I have never been able to get "X264" to work on my machine and instead have been using "mp4v" as the codec and ".mp4" for the container type in order to get an H.264 encoded output. After all of these issues are fixed, this is the result. Please try this and see if it works for you: import numpy as np import cv2 size = 720*16//9, 720 duration = 2 fps = 25 out = cv2.VideoWriter('output.mp4', cv2.VideoWriter_fourcc(*'mp4v'), fps, (size[1], size[0]), False) for _ in range(fps * duration): data = np.random.randint(0, 256, size, dtype='uint8') out.write(data) out.release()
Question: I'm doign an app with the Spotify API. My problem is I'm trying to get an access_token but It's not working. In the docs it says i need to send the body encoded as application/x-www-form-urlencoded so I search a little bit and It should work just setting request_body as a dictionary. This is the code of my function:  def get_access_token(self): auth_code, code_verifier = self.get_auth_code() endpoint = "https://accounts.spotify.com/api/token" # as docs said data should be encoded as application/x-www-form-urlencoded # as internet says i just need to send it as a dictionary. However it's not working request_body = { "client_id": f"{self.client_ID}", "grant_type": "authorization_code", "code": f"{auth_code}", "redirect_uri": f"{self.redirect_uri}", "code_verifier": f"{code_verifier}" } response = requests.post(endpoint, data=request_body) print(response)  The response I'm getting is always <Response [400]> Here are the docs, step 4 https://developer.spotify.com/documentation/general/guides/authorization-guide/#authorization-code-flow-with-proof-key-for-code-exchange-pkce NOTE: I tryed executing this as a curl and it works fine I'm not sure what I'm doing wrong in the python code here's the command: curl -d client_id={self.client_ID} -d grant_type=authorization_code -d code={auth_code} -d redirect_uri={self.redirect_uri} -d code_verifier={code_verifier} https://accounts.spotify.com/api/token 

Answer: You can specify the request type in the request header.  headers = {'Content-Type': 'application/x-www-form-urlencoded'} response = requests.post(endpoint, data=request_body, headers=headers) print(response)
Question: I am working on developing a python package. I use pip freeze > requirements.txt to add the required package into the requirement.txt file. However, I realized that some of the packages, instead of the package version, have some path in front of them. numpy==1.19.0 packaging==20.4 pandas @ file:///opt/concourse/worker/volumes/live/38d1301c-8fa9-4d2f-662e-34dddf33b183/volume/pandas_1592841668171/work pandocfilters==1.4.2  Whereas, inside the environment, I get: >>> pandas.__version__ '1.0.5'  Do you have any idea how to address this problem?

Answer: It looks like this is an open issue with pip freeze in version 20.1, the current workaround is to use: pip list --format=freeze > requirements.txt  In a nutshell, this is caused by changing the behavior of pip freeze to include direct references for distributions installed from direct URL references. You can read more about the issue on GitHub: pip freeze does not show version for in-place installs Output of "pip freeze" and "pip list --format=freeze" differ for packages installed via Direct URLs Better freeze of distributions installed from direct URL references
Question: I am using AWS Lambda and the functions I need to deploy require many different packages. Using serverless-python-requirements the zip file that is generated is 169.5MB, far greater than the 50MB limit. I have tried using Lambda Layers, but this doesn't solve the size issue. I have also tried dumping the zip file in an s3 bucket, but it is still too large to load when invoking the function. I need all of these packages and I'm not sure how I can deploy them all. My requirements.txt file looks like: bs4==0.0.1 gensim==3.8.3 matplotlib==3.2.2 nltk==3.5 numpy==1.19.0 openpyxl==3.0.4 pandas==1.0.5 pyLDAvis==2.1.2 spacy==2.3.1 XlsxWriter==1.2.9 

Answer: Very recently, AWS announced the support of EFS for Lambda.Read the announcement here. EFS or the Elastic File System is the NFS file system for compute nodes. Read more about them here. With this now you can essentially attach a network storage to your lambda function. I have personally used it to load huge reference files which are over the limit of Lambda's file storage. For a walkthrough you can refer to this article by AWS. To pick the conclusion from the article:  EFS for Lambda allows you to share data across function invocations, read large reference data files, and write function output to a persistent and shared store. After configuring EFS, you provide the Lambda function with an access point ARN, allowing you to read and write to this file system. Lambda securely connects the function instances to the EFS mount targets in the same Availability Zone and subnet.
Question: I tried to make a MySQL connection with peewee and followed the tutorial from their website: peewee quickstart So my code is the following: from peewee import * db = MySQLDatabase( host='127.0.0.1', user='root', password='', database='db_test' ) class Person(Model): name = CharField() birthday = DateField() class Meta: database = db class Pet(Model): owner = ForeignKeyField(Person, backref='pets') name = CharField() animal_type = CharField() class Meta: database = db db.connect() db.create_tables([Person, Pet]) db.close()  (My Database is from xampp) But when i execute this code I get this error message:  peewee.ImproperlyConfigured: MySQL driver not installed!  I tried to fix this by installing this MySQL Driver. But this changed absolutely nothing. Due to me beeing new to python I have no idea what I can do to fix this, if i'm just missing a import or if I have to install a library with pip?

Answer: The docs are clear, as is the error message: http://docs.peewee-orm.com/en/latest/peewee/database.html#using-mysql Install pymysql or mysqldb. To use the non-standard mysql-connector driver, you need to import the playhouse.mysql_ext module and use the MySQLConnectorDatabase implementation: http://docs.peewee-orm.com/en/latest/peewee/playhouse.html#mysql-ext