eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: On Windows 10, I am trying to save a Jupyter notebook as a pdf, under a name that will change for every run of the notebook. Here is what I have so far: name1 = 'July' name2 = 'August' jupyter_nb_filename = '{}_vs_{}'.format(name1,name2) !jupyter nbconvert --output-dir="C:\\mydir\\" --output=jupyter_nb_filename --to pdf --TemplateExporter.exclude_input=True mynotebook.ipynb   Current behaviour: Jupyter notebook is stored as "jupyter_nb_filename.pdf" Desired behaviour. Jupyter notebook to be stored as "July_vs_August.pdf"  I looked in the documentation but could not figure it out. Any help would be greatly appreciated.

Answer: You have to prefix the variable name with $ for it to be interpreted as a variable in a console command. See this question. !jupyter nbconvert --output-dir="C:\\mydir\\" --output=$jupyter_nb_filename --to pdf --TemplateExporter.exclude_input=True mynotebook.ipynb
Question: I'm trying to make a Lambda function using the AWS CDK They make it seem simple enough, but when I use cdk synth, it's giving me an error that the asset doesn't exist (even though it does exist). Here's my code: cwd = os.getcwd() aws_lambda.Function(self, "lambda_function", runtime=aws_lambda.Runtime.PYTHON_3_9, handler="index.handler", code=aws_lambda.Code.from_asset(os.path.join(cwd, "lambda_functions/lambda")) )   The file exists, and the error message prints the directory I expect it to, so what's the issue here?

Answer: From documentation The Code.from_asset(...) requires you to specify a directory or a .zip file. From your code you're referencing a directory which is not true. Change the path to add .zip extension. cwd = os.getcwd() aws_lambda.Function(self, "lambda_function", runtime=aws_lambda.Runtime.PYTHON_3_9, handler="index.handler", code=aws_lambda.Code.from_asset(os.path.join(cwd, "lambda_functions/lambda.zip")) )
Question: I'm new to using Pydantic and I'm using it to set up the models for FastAPI to integrate with my postgres database. I want to make a model that has an updated_at and created_at field which store the last datetime the model was updated and the datetime the model was created. I figured created_at could be something like this: created_at: datetime = datetime.now()  How would I do an updated_at so it updates the datetime automatically every time the model is updated?

Answer: You can use a validator which will update the field updated_at each time when some other data in the model will change. The root_validator and the validate_assignment config attribute are what you are looking for. This is the sample code: from datetime import datetime from time import sleep from pydantic import BaseModel,root_validator class Foo(BaseModel): data: str = "Some data" created_at: datetime = datetime.now() updated_at: datetime = datetime.now() class Config: validate_assignment = True @root_validator def number_validator(cls, values): values["updated_at"] = datetime.now() return values if __name__ == '__main__': bar = Foo() print(bar.dict()) sleep(5) bar.data = "New data" print(bar.dict())  and the output: { 'data': 'Some data', 'created_at': datetime.datetime(2022, 7, 31, 10, 41, 13, 176243), 'updated_at': datetime.datetime(2022, 7, 31, 10, 41, 13, 179253) } { 'data': 'New data', 'created_at': datetime.datetime(2022, 7, 31, 10, 41, 13, 176243), 'updated_at': datetime.datetime(2022, 7, 31, 10, 41, 18, 184983) }  You can see that there is a difference in microseconds after object creation. If this is a problem you can set:  updated_at: Optional[datetime] = None  and modify the validator like this:  @root_validator def number_validator(cls, values): if values["updated_at"]: values["updated_at"] = datetime.now() else: values["updated_at"] = values["created_at"] return values  and the new output: { 'data': 'Some data', 'created_at': datetime.datetime(2022, 7, 31, 10, 54, 33, 715379), 'updated_at': datetime.datetime(2022, 7, 31, 10, 54, 33, 715379) } { 'data': 'New data', 'created_at': datetime.datetime(2022, 7, 31, 10, 54, 33, 715379), 'updated_at': datetime.datetime(2022, 7, 31, 10, 54, 38, 728778) }
Question: Question: How can I disconnect, then reconnect to a vscode dev container without killing the ipynb kernel within my workspace? Background: I access my jupyter notebook inside a vscode dev container in order to have reproducibility of my project-specific environment. I connect to the container host machine on my laptop. Upon re-opening my vscode workspace after reconnecting to the container, my ipynb kernel is dead and all notebook computation must be repeated.

Answer: Try to use jupyter server instead. You can refer to this issue aout using the 'remote' server to control your kernel lifetime for details.
Question: I have this data set import pandas as pd import plotly.express as px elements = pd.DataFrame(data={"Area": ["A", "A", "A", "B", "B", "C", "C", "C"], "Branch": ["a1", "f55", "j23", "j99", "ci2", "p21", "o2", "q35"], "Good": [68, 3, 31, 59, 99, 86, 47, 47], "Neutral": [48, 66, 84, 4, 83, 76, 6, 89],"Bad": [72, 66, 50, 83, 29, 54, 84, 55]}) Area Branch Good Neutral Bad 0 A a1 68 48 72 1 A f55 3 66 66 2 A j23 31 84 50 3 B j99 59 4 83 4 B ci2 99 83 29 5 C p21 86 76 54 6 C o2 47 6 84 7 C q35 47 89 55  and i'm trying to plot it and get something that looks like this  stacked and grouped with labels, so I tried this fig_elements = px.bar(elements, x='Branch', y=["Good", "Neutral", "Bad"], orientation="v", template="plotly_dark", facet_col='Area') fig_elements.update_layout(plot_bgcolor="rgba(0,0,0,0)", xaxis=(dict(showgrid=False)), yaxis=(dict(showgrid=False)), barmode="stack")   can't add labels and on the bars, and areas have all branches not only the branches that belong to it, how can I fix that and add the labels?

Answer: import plotly.graph_objects as go df = elements.melt(id_vars=['Area', 'Branch'], value_vars=['Good', 'Neutral', 'Bad'], var_name='Rating') df ### Area Branch Rating value 0 A a1 Good 68 1 A f55 Good 3 2 A j23 Good 31 3 B j99 Good 59 4 B ci2 Good 99 5 C p21 Good 86 6 C o2 Good 47 7 C q35 Good 47 8 A a1 Neutral 48 9 A f55 Neutral 66 10 A j23 Neutral 84 11 B j99 Neutral 4 12 B ci2 Neutral 83     Plot rating_color = ['#17BEBB', '#FFD97D', '#EE6055'] fig = go.Figure() fig.update_layout(barmode='stack', width=900, height=800, uniformtext_minsize=8, uniformtext_mode='hide') for r, c in zip(df['Rating'].unique(), rating_color): df_plot = df[df['Rating'] == r] fig.add_trace( go.Bar( x=[df_plot['Area'], df_plot['Branch']], y=df_plot['value'], marker_color=c, name=r, text=df_plot['value'], width=0.45, textposition='auto') ) fig.show()
Question: I'm trying to type the following wrapper function. It takes another function and the function's arguments and runs it with some side effect. from typing import Callable, ParamSpec, TypeVar P = ParamSpec("P") R = TypeVar("R") def wrapper(func: Callable[P, R], *args: P.args, **kwargs: P.kwargs) -> R: # Run some side effects return func(*args, **kwargs) def f(a: int, b: int) -> int: return a + b # Prints 3 with the side effect. print(wrapper(f, 1, 2))  The above snippet works and mypy is happy. However, I don't like the black boxes that *args and **kwargs are and would like to make wrapper less generic. How do I type something similar to this: from typing import Callable, ParamSpec, TypeVar P = ParamSpec("P") R = TypeVar("R") # How do I express the fact that a and b are the input params of func? def wrapper(func: Callable[P, R], a: ?, b: ?) -> R: # Run some side effects return func(a, b) def f(a: int, b: int) -> int: return a + b # Mypy should be happy here! print(wrapper(f, 1, 2))  Is this possible?

Answer: When you know the arguments that will be passed to func, you don't need ParamSpec; ordinary TypeVars will do. AType = TypeVar('AType') BType = TypeVar('BType') def wrapper(func: Callable[[AType, BType], R], a: AType, b: BType) -> R: return func(a, b)  If there were additional arguments, then you would use Concatenate to combine the known arguments with ParamSpec. AType = TypeVar('AType') BType = TypeVar('BType') P = ParamSpec('P') def wrapper(func: Callable[Concatenate[AType, BType, P], R], a: AType, b: BType, *args: P.args, **kwargs: P.kwargs) -> R: return func(a, b, *args, **kwargs)
Question: I would like to plot two time series, one of which is in cftime and the other in datetime. One possibility is to convert cftime to datetime, but this might give strange results for nonstandard cftime calendars (e.g. NoLeap). As such, I am trying to convert the datetime to cftime. I can brute-force it as follows, but is there a built-in method available? >>> import pandas as pd >>> import xarray as xr >>> >>> da = xr.DataArray( ... [1, 2], coords={"time": pd.to_datetime(["2000-01-01", "2000-02-02"])}, dims=["time"] ... ) >>> print(da.time) <xarray.DataArray 'time' (time: 2)> array(['2000-01-01T00:00:00.000000000', '2000-02-02T00:00:00.000000000'], dtype='datetime64[ns]') Coordinates: * time (time) datetime64[ns] 2000-01-01 2000-02-02 >>> >>> >>> import cftime >>> >>> >>> def datetime_to_cftime(dates, kwargs={}): ... return [ ... cftime.datetime( ... date.dt.year, ... date.dt.month, ... date.dt.day, ... date.dt.hour, ... date.dt.minute, ... date.dt.second, ... date.dt.microsecond, ... **kwargs ... ) ... for date in dates ... ] ... >>> datetime_to_cftime(da.time) [cftime.datetime(2000, 1, 1, 0, 0, 0, 0, calendar='standard', has_year_zero=False), cftime.datetime(2000, 2, 2, 0, 0, 0, 0, calendar='standard', has_year_zero=False)] 

Answer: Indeed you might consider using DataArray.convert_calendar. For example if you would like to convert datetime64 values to cftime.DatetimeNoLeap objects, you could do something like this: >>> da.convert_calendar("noleap") <xarray.DataArray (time: 2)> array([1., 2.]) Coordinates: * time (time) object 2000-01-01 00:00:00 2000-02-02 00:00:00  This method is new as of xarray version 0.20.0.
Question: When I use the run directive in snakemake (using python code) it doesn't produce any kind of error message for troubleshooting. Is this desired behavior? Am I missing something? Here a minimal example using snakemake 7.8.3 and python 3.9.13. I invoked snakemake with the -p option which in shell directive outputs the exact code as passed to the shell (but doesn't do anything for run directive I guess). Snakefile: def useless_function(): return[thisVariableAlsoDoesntExist] rule all: input: "final.txt" rule test: output: "final.txt" run: print(thisVariableDoesNotExist) useless_function()  Stdout: Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Job stats: job count min threads max threads ----- ------- ------------- ------------- all 1 1 1 test 1 1 1 total 2 1 1 Select jobs to execute... [Mon Jul 25 18:59:13 2022] rule test: output: final.txt jobid: 1 reason: Missing output files: final.txt resources: tmpdir=/tmp Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2022-07-25T185913.188760.snakemake.log  Expected error message (when function and print command are executed directly on python console): >>> print(thisVariableDoesNotExist) Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'thisVariableDoesNotExist' is not defined >>> useless_function() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 2, in useless_function NameError: name 'thisVariableAlsoDoesntExist' is not defined 

Answer: I suspect you are hitting this recent bug https://github.com/snakemake/snakemake/issues/1698. If that is the case, you can downgrade to v7.6.2 or work around it, i.e. bear with it or wrap the the code in run in a self-contained script that you execute via shell. This latter is not a bad solution anyway since it keeps the code nicely isolated.
Question: I have a list and I would like to convert it to a pandas dataframe. In the second column, I want to give all zeros but I got "object of type 'int' has no len()" error. The thing I did is this: df = pd.DataFrame([all_equal_timestamps['A'], 0], columns=['data','label'])  How can i add second column with all zeros to this dataframe in the easiest manner and why did the code above give me this error?

Answer: Not sure what is in all_equal_timestamps, so I presume it's a list of elements. Do you mean to get this result? import pandas as pd all_equal_timestamps = {'A': ['1234', 'aaa', 'asdf']} df = pd.DataFrame(all_equal_timestamps['A'], columns=['data']).assign(label=0) # df['label'] = 0 print(df)  Output:  data label 0 1234 0 1 aaa 0 2 asdf 0  If you're creating a DataFrame with a list of lists, you'd expect something like this df = pd.DataFrame([ all_equal_timestamps['A'], '0'*len(all_equal_timestamps['A']) ], columns=['data', 'label', 'anothercol']) print(df)  Output:  data label anothercol 0 1234 aaa asdf 1 0 0 0
Question: I have a dataframe like this:  ID value repeat ratio 0 0 IDx10 6 0.5 1 1 IDx11 7 1.5 2 2 IDx12 8 2.5  and i have a list like this:  l = [1,2] What i want to do is to duplicate every row the number of times of the length of the list and in every new row put each value of the list. And getting a dataframe like this:  ID value repeat ratio value 0 0 IDx10 6 0.5 1 1 0 IDx10 6 0.5 2 2 1 IDx11 7 1.5 1 3 1 IDx11 7 1.5 2 4 2 IDx12 8 2.5 1 5 2 IDx12 8 2.5 2 

Answer: Let us do a cross merge: out = df.merge(pd.Series(l, name='value2'), how='cross')  output:  ID value repeat ratio value2 0 0 IDx12 6 0.5 1 1 0 IDx12 6 0.5 2 2 0 IDx12 6 0.5 3 3 1 IDx12 7 1.5 1 4 1 IDx12 7 1.5 2 5 1 IDx12 7 1.5 3 6 2 IDx12 8 2.5 1 7 2 IDx12 8 2.5 2 8 2 IDx12 8 2.5 3
Question: I want to type hint the return object of some_func function, which is always the same format. Is this correct? from typing import List, Dict def some_func() -> List[Dict[str, int, str, List[CustomObject]]]: my_list = [ {"ID": 1, "cargo": [CustomObject(), CustomObject()]}, {"ID": 2, "cargo": [CustomObject(), CustomObject()]}, {"ID": 2, "cargo": [CustomObject(), CustomObject()]} ] return my_list 

Answer: one way of correctly type hinting would look like this: from typing import List, Dict, Union def some_func() -> List[Dict[str, Union[int, List[CustomObject]]]]: my_list = [ {"ID": 1, "cargo": [CustomObject(), CustomObject()]}, {"ID": 2, "cargo": [CustomObject(), CustomObject()]}, {"ID": 2, "cargo": [CustomObject(), CustomObject()]} ] return my_list  for a brief explanation, when type annotating a dict, you can only have two arguments, the first being the type of any given key, and the second, the type of any given value. since keys are always strings, we keep Dict[str, and our keys can be either an integer, or of type CustomObject, to represent different possibilities in type annotations we use Union, and so in the end we get: Dict[str, Union[int, List[CustomObject]]  for some additional notes, in python 3.9 or above you may replace Dict imported from typing, with the builtin dict, and in python 3.10 or above, unions can be represented as type | type, and you can replace List with the builtin list for a cleaner result you may want to use a typedDict, the overall result would be: from typing import TypedDict class customDict(TypedDict): ID: int cargo: list[CustomObject] def some_func() -> list[customDict]: my_list = [ {"ID": 1, "cargo": [CustomObject(), CustomObject()]}, {"ID": 2, "cargo": [CustomObject(), CustomObject()]}, {"ID": 2, "cargo": [CustomObject(), CustomObject()]} ] return my_list  each property of the typed dict represents a key of the dicts in my_list, and the following reprents the type of value related to the key
Question: In stats models I have this code from statsmodels.tsa.arima.model import ARIMA from statsmodels.graphics.tsaplots import plot_predict df1.drop(df1.columns.difference(['PTS']), 1, inplace=True) model = ARIMA(df1.PTS, order=(0, 15,0)) res = model.fit() res.plot_predict(start='2021-10-19', end='2022-04-05') plt.show()  However when it goes to plt.show I get the ARIMA results object has no attribute plot predict. What do i do to fix this?

Answer: The .plot_predict() method no longer exists with the changes to the ARIMA classes in statsmodels version 13. So, just use the plot_predict() function that you already imported in your code. Here is an example with a different dataset: import matplotlib.pyplot as plt import pandas as pd import statsmodels.api as sm from statsmodels.graphics.tsaplots import plot_predict from statsmodels.tsa.arima.model import ARIMA dta = sm.datasets.sunspots.load_pandas().data[['SUNACTIVITY']] dta.index = pd.date_range(start='1700', end='2009', freq='A') res = ARIMA(dta, order=(0,2,0)).fit() fig, ax = plt.subplots() ax = dta.loc['1950':].plot(ax=ax) plot_predict(res, '1990', '2012', ax=ax) plt.show()
Question: I want to do a complex list comparison with python. I want to see if listB contains all of the items from listA and if they are in the same order. But I do not care if listB has extra items or interleaved items. Examples: listA = ['A','B','C','D','E'] listB = [':','A','*','B','C','D','E','`']  A, B, C, D, and E all appear in listB and are presented in the same order even though A and B have an item in between them and items at the start and end of listB. Extra complicated: listA = ['A','B','C','D','E'] listB = ['A','*','C','B','C','D','E']  A, B, C, D, and E all appear in listB and are presented in the same order even though A and B have two items in between them and one of those items happens to be something we are searching for. But since we are looking if A -> B is sequential and B -> C is sequential the fact that we also have C -> B -> C shouldn't matter. So, listA = ['A','B','C','D','E'] listB = [':','A','*','B','C','D','E','`']  Would be True listA = ['A','B','C','D','E'] listB = ['A','*','C','B','C','D','E']  Would be True But something like: listA = ['A','B','C','D','E'] listB = ['A','B','C','D','F']  or even listB = ['A','B','C','D']  Would be False If it get a False answer, I'd ideally like to be able to point to where the break in sequence happened -- i.e. E is missing.

Answer: Simple solution using a nested loop. Walk over listA and search the elements in listB in order. Should you fail at any point -> this is not a substring:  def check(listA, listB): start = 0 for a in listA: for i in range(start, len(listB)): if a == listB[i]: start = i+1 break else: # triggered only if no break # print(f'{a} not found after position {start}') return False return True check('ABCDE', 'A*CBCDE') # True check('ABCDEF', 'A*CBCDE') # False check ('', '') # True check('ABA', 'ABBA') # True  NB. Using strings here for clarity, but this works for any iterable. To get information on the non-found item, you can uncomment the print. Example: check('ABCA', 'ABBA') C not found after position 2 # False
Question: Hello everyone I'm attempting to use the Gmail API to print out specific emails from a sender. I've managed to do some research and watched some videos on how to get the sender and the subject printed off but for some reason, I cant get the body of the message to print off. I've looked through the Gmail API and haven't found anything to help with printing the body in text form. Any help with printing off the body of the email, please... service = build('gmail', 'v1', credentials=creds) results = service.users().messages().list(userId='me', labelIds=['INBOX'], q="from:specific email, is:unread").execute() messages = results.get('messages', []) if not messages: print("You have no New Messages.") else: message_count = 0 for message in messages: msg = service.users().messages().get(userId='me', id=message['id']).execute() message_count= message_count + 1 email_data= msg['payload']['headers'] for values in email_data: name = values["name"] if name == "From": from_name = values ["value"] print(from_name) subject= [j['value'] for j in email_data if j["name"]=="Subject"] print(subject)  This code like I said pulls the specific email and prints the sender, and the subject all I'm missing is the body. I've tried following what was posted in this stackoverflow: How to retrieve the whole message body using Gmail API (python) But I couldn't manage to get it to work

Answer: In your script, how about the following modification? Modified script: service = build("gmail", "v1", credentials=creds) results = service.users().messages().list(userId="me", labelIds=["INBOX"], q="from:specific email, is:unread").execute() messages = results.get("messages", []) if not messages: print("You have no New Messages.") else: message_count = 0 for message in messages: msg = service.users().messages().get(userId="me", id=message["id"]).execute() message_count = message_count + 1 email_data = msg["payload"]["headers"] for values in email_data: name = values["name"] if name == "From": from_name = values["value"] print(from_name) subject = [j["value"] for j in email_data if j["name"] == "Subject"] print(subject) # I added the below script. for p in msg["payload"]["parts"]: if p["mimeType"] in ["text/plain", "text/html"]: data = base64.urlsafe_b64decode(p["body"]["data"]).decode("utf-8") print(data)   In this case, please add import base64.  When this script is run, both the text body and the HTML body are retrieved. For example, when you want to retrieve only the text body, please modify if p["mimeType"] in ["text/plain", "text/html"]: to if p["mimeType"] == "text/plain":.   Reference:  Method: users.messages.get
Question: I am using the below script to try and separate handwritten text from the lines which the text was written on. Currently I am trying to select the lines. This seems to work well when the line are solid but when the lines are a string of dots it becomes tricky. To try and get around this I have tried using dilate to make the dots into solid lines, but dilate is also making the text solid which then gets pick up as horizontal lines. I can tweak the kernel for each image but that is not a workable solution when dealing with thousandths of images. Can someone suggest how I might make this work please. Is this the best approach or is there a better approach for selecting these lines? Sample images    import cv2 file_path = r'image.jpg' image = cv2.imread(file_path) # resize image if image is bigger then screen size print('before Dimensions : ', image.shape) if image.shape[0] > 1200: image = cv2.resize(image, None, fx=0.2, fy=0.2) print('after Dimensions : ', image.shape) result = image.copy() gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY) thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1] # Applying dilation to make lines solid kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3,3)) dilation = cv2.dilate(thresh, kernel, iterations = 1) # Detect horizontal lines horizontal_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (40,1)) detect_horizontal = cv2.morphologyEx(dilation, cv2.MORPH_OPEN, horizontal_kernel, iterations=2) cnts = cv2.findContours(detect_horizontal, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) cnts = cnts[0] if len(cnts) == 2 else cnts[1] for c in cnts: cv2.drawContours(result, [c], -1, (36,255,12), 2) cv2.imshow('1- gray', gray) cv2.imshow("2- thresh", thresh) cv2.imshow("3- detect_horizontal", detect_horizontal) cv2.imshow("4- result", result) cv2.waitKey(0) cv2.destroyAllWindows() 

Answer: By finding contours, we can eliminate smaller ones by their area using cv2.contourArea. This will work under the assumption that the image contains dotted lines. Code: # read image, convert to grayscale and apply Otsu threshold img = cv2.imread('text.jpg') gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY) th = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1] # create black background of same image shape black = np.zeros((img.shape[0], img.shape[1], 3), np.uint8) # find contours from threshold image contours = cv2.findContours(th, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[0] # draw contours whose area is above certain value area_threshold = 7 for c in contours: area = cv2.contourArea(c) if area > area_threshold: black = cv2.drawContours(black,[c],0,(255,255,255),2)  black:  To refine this for more images, you can filter contours using some statistical measures (like mean, median, etc.)