eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have stock market data for a single security going back 20 years. The data is currently in an Pandas DataFrame, in the following format:  The problem is, I do not want any "after hours" trading data in my DataFrame. The market in question is open from 9:30AM to 4PM (09:30 to 16:00 on each trading day). I would like to drop all rows of data that are not within this time frame.  My instinct is to use a Pandas mask, which I know how to do if I wanted certain hours in a single day: mask = (df['date'] > '2015-07-06 09:30:0') & (df['date'] <= '2015-07-06 16:00:0') sub = df.loc[mask]  However, I have no idea how to use one on a revolving basis to remove the data for certain times of day over a 20 year period.

Answer: Problem here is how you are importing data. There is no indicator whether 04:00 is am or pm? but based on your comments we need to assume it is PM. However input is showing it as AM. To solve this we need to include two conditions with OR clause.   9:30-11:59 0:00-4:00  Input: df = pd.DataFrame({'date': {880551: '2015-07-06 04:00:00', 880552: '2015-07-06 04:02:00',880553: '2015-07-06 04:03:00', 880554: '2015-07-06 04:04:00', 880555: '2015-07-06 04:05:00'}, 'open': {880551: 125.00, 880552: 125.36,880553: 125.34, 880554: 125.08, 880555: 125.12}, 'high': {880551: 125.00, 880552: 125.36,880553: 125.34, 880554: 125.11, 880555: 125.12}, 'low': {880551: 125.00, 880552: 125.32,880553: 125.21, 880554: 125.05, 880555: 125.12}, 'close': {880551: 125.00, 880552: 125.32,880553: 125.21, 880554: 125.05, 880555: 125.12}, 'volume': {880551: 141, 880552: 200,880553: 750, 880554: 17451, 880555: 1000}, }, ) df.head() date open high low close volume 880551 2015-07-06 04:00:00 125.00 125.00 125.00 125.00 141 880552 2015-07-06 04:02:00 125.36 125.36 125.32 125.32 200 880553 2015-07-06 04:03:00 125.34 125.34 125.21 125.21 750 880554 2015-07-06 04:04:00 125.08 125.11 125.05 125.05 17451 880555 2015-07-06 04:05:00 125.12 125.12 125.12 125.12 1000 from datetime import time start_first = time(9, 30) end_first = time(11, 59) start_second = time(0, 00) end_second = time(4,00) df['date'] = pd.to_datetime(df['date']) df= df[(df['date'].dt.time.between(start_first, end_first)) | (df['date'].dt.time.between(start_second, end_second))] df date open high low close volume 880551 2015-07-06 04:00:00 125.0 125.0 125.0 125.0 141  Above is not good practice, and I strongly discourage to use this kind of ambiguous data. long time solution is to correctly populate data with am/pm. We can achieve it in two way in case of correct data format: 1) using datetime from datetime import time start = time(9, 30) end = time(16) df['date'] = pd.to_datetime(df['date']) df= df[df['date'].dt.time.between(start, end)]  2) using between time, which only works with datetime index df['date'] = pd.to_datetime(df['date']) df = (df.set_index('date') .between_time('09:30', '16:00') .reset_index())  If you still face error, edit your question with line by line approach and exact error.
Question: In django abstract classes seem to be posibble by using: class Meta: abstract = True  However I do not see how to declare abstract methods/functions within these classes that do not contain any logic like e.g. class AbstractClass(models.Model): def abstractFunction(): class Meta: abstract = True  The library abc repectively the notation @abstractmethod doesnt seem to applicable here, or am I wrong?

Answer: From what I understand, you're correct. Meta: abstract=True in Django models is there to make sure Django doesn't create database tables for the model. It predates Pythons ABC, which might explain the naming/functional confusion. Inheriting from both ABC and django.db.models.Model raises metaclass exception However, a quick search gave me this. Don't know if it breaks anything. https://gist.github.com/gavinwahl/7778717 I do like how ABC raises an exception at class instantiation, so it would be pretty neat. Let me know if you find it useful.
Question: Question How to shade or colorize the background of a seaborn plot using a column of a dataframe? Code snippet import numpy as np import seaborn as sns; sns.set() import matplotlib.pyplot as plt fmri = sns.load_dataset("fmri") fmri.sort_values('timepoint',inplace=True) ax = sns.lineplot(x="timepoint", y="signal", data=fmri) arr = np.ones(len(fmri)) arr[:300] = 0 arr[600:] = 2 fmri['background'] = arr ax = sns.lineplot(x="timepoint", y="signal", hue="event", data=fmri)  Which produced this graph:  Desired output What I'd like to have, according to the value in the new column 'background' and any palette or user defined colors, something like this: 

Answer: ax.axvspan() could work for you, assuming backgrounds don't overlap over timepoints. import numpy as np import seaborn as sns; sns.set() import matplotlib.pyplot as plt fmri = sns.load_dataset("fmri") fmri.sort_values('timepoint',inplace=True) arr = np.ones(len(fmri)) arr[:300] = 0 arr[600:] = 2 fmri['background'] = arr fmri['background'] = fmri['background'].astype(int).astype(str).map(lambda x: 'C'+x) ax = sns.lineplot(x="timepoint", y="signal", hue="event", data=fmri) ranges = fmri.groupby('background')['timepoint'].agg(['min', 'max']) for i, row in ranges.iterrows(): ax.axvspan(xmin=row['min'], xmax=row['max'], facecolor=i, alpha=0.3)
Question: if session['dp'] := current_user.avatar : ^ SyntaxError: cannot use assignment expressions with subscript  Why Python forbids this use of walrus operator?

Answer: Because, as the alternative name (named expressions) suggests, the left hand side of the walrus operator is to be a NAME. Therefore, by definition such expressions as noted in your question as well as, for instance, function calls are not allowed to be assigned in this form. The documentation also specifies:  Single assignment targets other than a single NAME are not supported  To further this argument, one can notice that cPython explicitly checks if the expression is Name_kind: if (target->kind != Name_kind) { const char *expr_name = get_expr_name(target); if (expr_name != NULL) { ast_error(c, n, "cannot use assignment expressions with %s", expr_name); } return NULL; }
Question: I'm working on a Django project on which i have a queryset of a 'A' objects ( A.objects.all() ), and i need to annotate multiple fields from a 'B' objects' Subquery. The problem is that the annotate method can only deal with one field type per parameter (DecimalField, CharField, etc.), so, in order to annotate multiple fields, i must use something like: A.objects.all().annotate(b_id =Subquery(B_queryset.values('id')[:1], b_name =Subquery(B_queryset.values('name')[:1], b_other_field =Subquery(B_queryset.values('other_field')[:1], ... )  Which is very inefficient, as it creates a new subquery/subselect on the final SQL for each field i want to annotate. I would like to use the same Subselect with multiple fields on it's values() params, and annotate them all on A's queryset. I'd like to use something like this: b_subquery = Subquery(B_queryset.values('id', 'name', 'other_field', ...)[:1]) A.objects.all().annotate(b=b_subquery)  But when i try to do that (and access the first element A.objects.all().annotate(b=b_subquery)[0]) it raises an exception: {FieldError}Expression contains mixed types. You must set output_field. And if i set Subquery(B_quer...[:1], output_field=ForeignKey(B, models.DO_NOTHING)), i get a DB exception: {ProgrammingError}subquery must return only one column In a nutshell, the whole problem is that i have multiple Bs that "belongs" to a A, so i need to use Subquery to, for every A in A.objects.all(), pick a specific B and attach it on that A, using OuterRefs and a few filters (i only want a few fields of B), which seens a trivial problem for me. Thanks for any help in advance!

Answer: What I do in such situations is to use prefetch-related  a_qs = A.objects.all().prefetch_related( models.Prefetch('b_set', # NOTE: no need to filter with OuterRef (it wont work anyway) # Django automatically filter and matches B objects to A queryset=B_queryset, to_attr='b_records' ) )  Now a.b_records will be a list containing a's related b objects. Depending on how you filter your B_queryset this list may be limited to only 1 object.
Question: If I don't use await to call the async function, I will get back a coroutine. In that case, what will happen for the coroutine? Do I have to manually execute the coroutine? Or this coroutine will continue to run itself in the background?  Using await async def work(): result = await stuff()  Without await async def work(): result = stuff() 

Answer: From the official docs:  Note that simply calling a coroutine will not schedule it to be executed:  That means you did not call your function actually so there no one waiting for anything and nothing to be waiting for if you did not place await before your function call. You could instead schedule a task for it or many tasks using asyncio: import asyncio async def main(): loop = asyncio.get_event_loop() t1 = loop.create_task(stuff()) t2 = loop.create_task(stuff()) loop = asyncio.get_event_loop() loop.run_until_complete(main())  To find more about this, I would recommend reading https://docs.python.org/3/library/asyncio-task.html
Question: I have the following pandas dataframe df : import pandas as pd from io import StringIO s = '''\ "Unnamed: 0","Unnamed: 1" Objet,"Unités vendues" Chaise,3 Table,2 Tabouret,1 ''' df = pd.read_csv(StringIO(s))  which looks as:  Unnamed: 0 Unnamed: 1 0 Objet Unités vendues 1 Chaise 3 2 Table 2 3 Tabouret 1  My target is to make the first row as header. I use : headers = df.iloc[0] df.columns = [headers]  However, the "0" appears in index column name (which is normal, because this 0 was in the first row). 0 Objet Unités vendues 1 Chaise 3 2 Table 2  I tried to delete it in many way, but nothing work : Neither del df.index.name from this post Neither df.columns.name = None from this post or this one (which is the same situation) How can I have this expected output :  Objet Unités vendues 1 Chaise 3 2 Table 2 

Answer: what about defining that when you load your table in the first place?  pd.read_csv('filename', header = 1) otherwise I guess you can just do this: df.drop('0', axis = 1)
Question: I have such a method: def select_unassigned_variable(self, variables: List[V]) -> Optional(V): I want this to return something of type V or None in some cases. But I get such error: TypeError: Cannot instantiate typing.Optional What should I change?

Answer: You need to use it with brackets instead of parentheses: def select_unassigned_variable(self, variables: List[V]) -> Optional[V]:  like you did with List.
Question: This must be a super trivial issue, but i've updated my windows virtual machine with; pip install --upgrade google-cloud-storage  However, when I run the script I still receive the following error; Traceback (most recent call last): File "file.py", line 6, in <module> from google.cloud import bigquery, storage ImportError: cannot import name 'bigquery'  Any suggestions or workarounds? Thanks, Neel R

Answer: From a fresh setup of the VM; Failed - pip3 install --upgrade google-cloud Worked - pip3 install --upgrade google-cloud-bigquery Worked - pip3 install --upgrade google-cloud-storage It appears that individual product solutions should be installed instead of the generic google-cloud. If you're still stuck, this helped!
Question: When using reticulate package in order to use Python inside R, we can create a virtualenv thanks to the command reticulate::virtualenv_create specifying env name and the path to the python bin. We can also add packages to the previously created environment like this:  reticulate::virtualenv_create(envname = 'venv_shiny_app', python = '/usr/bin/python3') reticulate::virtualenv_install('venv_shiny_app', packages = c('numpy', 'xlrd', 'pandas', 'beautifulsoup4', 'joblib'))  Is it possible to install a specific version of those packages?? Thanks

Answer: You can request a specific version of a package with, for example: reticulate::virtualenv_install(packages = c("numpy==1.8.0"))
Question: I was wondering how to calculate the WOE and IV in python. Are there any dedication function in numpy/scipy/pandas/sklearn? Here is my example dataframe: import numpy as np import pandas as pd np.random.seed(100) df = pd.DataFrame({'grade': np.random.choice(list('ABCD'),size=(20)), 'pass': np.random.choice([0,1],size=(20)) }) df 

Answer: Formulas for woe and iv:  Code to achieve this: import numpy as np import pandas as pd np.random.seed(100) df = pd.DataFrame({'grade': np.random.choice(list('ABCD'),size=(20)), 'pass': np.random.choice([0,1],size=(20)) }) feature,target = 'grade','pass' df_woe_iv = (pd.crosstab(df[feature],df[target], normalize='columns') .assign(woe=lambda dfx: np.log(dfx[1] / dfx[0])) .assign(iv=lambda dfx: np.sum(dfx['woe']* (dfx[1]-dfx[0])))) df_woe_iv  output pass 0 1 woe iv grade A 0.3 0.3 0.000000 0.690776 B 0.1 0.1 0.000000 0.690776 C 0.2 0.5 0.916291 0.690776 D 0.4 0.1 -1.386294 0.690776
Question: I created my model in pytorch and is working really good, but when i want to test just one image batch_size=1 always return the second class (in this case a dog). I tried to test with batch > 1 and in all cases this works! The architecture: model = models.densenet121(pretrained=True) for param in model.parameters(): param.requires_grad = False from collections import OrderedDict classifier = nn.Sequential(OrderedDict([ ('fc1', nn.Linear(1024, 500)), ('relu', nn.ReLU()), ('fc2', nn.Linear(500, 2)), ('output', nn.LogSoftmax(dim=1)) ])) model.classifier = classifier  so my tensors are [batch, 3, 224, 224] i have tried with: resize reshape unsqueeze(0)  the response when is one image is always [[0.4741, 0.5259]] My Test Code from PIL import * msize = 256 loader = transforms.Compose([transforms.Scale(imsize), transforms.ToTensor()]) def image_loader(image_name): """load image, returns cuda tensor""" image = Image.open(image_name) image = loader(image).float() image = image.unsqueeze(0) return image.cuda() image = image_loader('Cat_Dog_data/test/cat/cat.16.jpg') with torch.no_grad(): logits = model.forward(image) ps = torch.exp(logits) _, predTest = torch.max(ps,1) print(ps) ## same value in all cases imagen_mostrar = images[ii].to('cpu') helper.imshow(imagen_mostrar,title=clas_perro_gato(predTest), normalize=True)  Second Test Code andrea_data = datasets.ImageFolder(data_dir + '/andrea', transform=test_transforms) andrealoader = torch.utils.data.DataLoader(andrea_data, batch_size=1, shuffle=True) dataiter = iter(andrealoader) images, labels = dataiter.next() images, labels = images.to(device), labels.to(device) ps = torch.exp(model.forward(images)) _, predTest = torch.max(ps,1) print(ps.float())  if i changed my batch_size to 1 always returned a tensor who say that is a dog [0.43,0.57] for example. Thanks!

Answer: I realized that my model wasn't in eval mode. So i just added model.eval() and now that's all, works for any size batch
Question: I have a dataframe that contains data I want to upload into MongoDB. Below is the data:  MongoRow = pd.DataFrame.from_dict({'school': {1: schoolID}, 'student': {1: student}, 'date': {1: dateToday}, 'Probability': {1: probabilityOfLowerThanThreshold}}) school student date Probability 1 5beee5678d62101c9c4e7dbb 5bf3e06f9a892068705d8420 2020-03-27 0.000038  I have the following code which checks if a row in mongo contains the same student ID and date, if it doesn't then it adds the row: def getPredictions(school): schoolDB = DB[school['database']['name']] schoolPredictions = schoolDB['session_attendance_predicted'] Predictions = schoolPredictions.aggregate([{ '$project': { 'school': '$school', 'student':'$student', 'date':'$date' } }]) return list(Predictions) Predictions = getPredictions(school) Predictions = pd.DataFrame(Predictions) schoolDB = DB[school['database']['name']] collection = schoolDB['session_attendance_predicted'] import json for i in Predictions.index: schoolOld = Predictions.loc[i,'school'] studentOld = Predictions.loc[i,'student'] dateOld = Predictions.loc[i,'date'] if(studentOld == student and date == dateOld): print("Student Exists") #UPDATE THE ROW WITH NEW VALUES else: print("Student Doesn't Exist") records = json.loads(df.T.to_json()).values() collection.insert(records)  However if it does exist, I want it to update the row with the new values. Does anyone know how to do this? I have looked at pymongo upsert but I'm not sure how to use it. Can anyone help? '''''''UPDATE''''''' The above is partly working now, however, I am now getting an error with the following code: dateToday = datetime.datetime.combine(dateToday, datetime.time(0, 0)) MongoRow = pd.DataFrame.from_dict({'school': {1: schoolID}, 'student': {1: student}, 'date': {1: dateToday}, 'Probability': {1: probabilityOfLowerThanThreshold}}) data_dict = MongoRow.to_dict() for i in Predictions.index: print(Predictions) collection.replace_one({'student': student, 'date': dateToday}, data_dict, upsert=True)  Error: InvalidDocument: documents must have only string keys, key was 1 

Answer: To upsert you cannot use insert() (deprecated) insert_one() or insert_many(). You must use one of the collection level operators that supports upserting. To get started I would point you towards reading the dataframe line by line and using replace_one() on each line. There are more advanced ways of doing this but this is the easiest.  Your code will look a bit like: collection.replace_one({'Student': student, 'Date': date}, record, upsert=True)
Question: I have a df:  id timestamp data group Date 27001 27242 2020-01-01 09:07:21.277 19.5 1 2020-01-01 27002 27243 2020-01-01 09:07:21.377 19.0 1 2020-01-01 27581 27822 2020-01-02 07:53:05.173 19.5 1 2020-01-02 27582 27823 2020-01-02 07:53:05.273 20.0 1 2020-01-02 27647 27888 2020-01-02 10:01:46.380 20.5 1 2020-01-02 ...  and I would like to calculate the time difference between row 1 and row 2 in seconds. I could do it with  df['timediff'] = (df['timestamp'].shift(-1) - df['timestamp']).dt.total_seconds()  However, when I zoom in to look at only 2 rows, ie. row1 and row0, with code: difference = (df.loc[0, 'timestamp'] - df.loc[1, 'timestamp']).dt.total_seconds()  it returned error   AttributeError: 'Timedelta' object has no attribute 'dt'  Why is this happening?

Answer: As @hpaulj said in the comments, dt is only associated with dataframe like object. So to obtain total seconds you have to use difference = (df.loc[0, 'timestamp'] - df.loc[1, 'timestamp']).total_seconds()
Question: In the tutorial for setting up the aesthetics of your plots, there are a few different methods:  set_style set_context axes_style  Each one of these accepts an rc keyword parameter dictionary. In each individual API page for the above three functions, it says:  rcdict, optional: Parameter mappings to override the values in the preset seaborn style dictionaries. This only updates parameters that are considered part of the style definition.  Back in the tutorial page, under axes_style it goes on to say exactly how you can see what parameters are available for the rc dictionary for this one function:  If you want to see what parameters are included, you can just call the function with no arguments, which will return the current settings:  However, using this on the other functions always returns None. So, for example, I am using the following mix of matplotlib and seaborn to set parameters: mpl.rcParams['figure.figsize'] = [16,10] viz_dict = { 'axes.titlesize':18, 'axes.labelsize':16, } sns.set_context("notebook", rc=viz_dict) sns.set_style("whitegrid")  I also noticed that putting my dictionary in the set_style method does nothing, while, at least for those parameters, it only works in set_context. This means that they each have mutually exclusively characteristics that can be edited. However, this is not outlined anywhere in the docs.  I want to know which one of these three functions will accept a parameter for figsize. I'd also be curious to see what else they accept that might help me fine-tune things. My goal is to exclusively use the seaborn interface as often as possible. I don't need the fine tune control of things matplotlib provides, and often find it awkward anyway. 

Answer: It would appear that the answer is 'none of the above'. The valid keys for set_style and set_context are listed here: _style_keys = [ "axes.facecolor", "axes.edgecolor", "axes.grid", "axes.axisbelow", "axes.labelcolor", "figure.facecolor", "grid.color", "grid.linestyle", "text.color", "xtick.color", "ytick.color", "xtick.direction", "ytick.direction", "lines.solid_capstyle", "patch.edgecolor", "patch.force_edgecolor", "image.cmap", "font.family", "font.sans-serif", "xtick.bottom", "xtick.top", "ytick.left", "ytick.right", "axes.spines.left", "axes.spines.bottom", "axes.spines.right", "axes.spines.top",] _context_keys = [ "font.size", "axes.labelsize", "axes.titlesize", "xtick.labelsize", "ytick.labelsize", "legend.fontsize", "axes.linewidth", "grid.linewidth", "lines.linewidth", "lines.markersize", "patch.linewidth", "xtick.major.width", "ytick.major.width", "xtick.minor.width", "ytick.minor.width", "xtick.major.size", "ytick.major.size", "xtick.minor.size", "ytick.minor.size",]  Also note that set_style is just a convenience function which calls axes_style. So you will have to use matplotlib.rcParams, although if the typical rcParams['figure.figsize'] = [16,10] syntax is not amenable you could of course create your own style.
Question: i have short code for crop image all image in folder that i labeled and save as csv using opencv like this: import os, sys from PIL import Image import cv2 import pandas as pd # The annotation file consists of image names, text label, # bounding box information like xmin, ymin, xmax and ymax. ANNOTATION_FILE = 'data/annot_crop_plate.csv' df = pd.read_csv(ANNOTATION_FILE) #image directory path IMG_DIR = 'data/images' # The cropped images will be stored here CROP_DIR = 'data/crops' files = df['filename'] size = (200,200) for file in files: print(file) img = cv2.imread(IMG_DIR +'/' + file) annot_data = df[df['filename'] == file] xmin = int(annot_data['xmin']) ymin = int(annot_data['ymin']) xmax = int(annot_data['xmax']) ymax = int(annot_data['ymax']) crop = img[ymin:ymax,xmin:xmax] new_crop = cv2.resize(crop, dsize=size, interpolation=cv2.INTER_CUBIC) new_crop.save(CROP_DIR + '/' + file.split('.')[0] + '.png', 'PNG', quality=90)  but in the end of line have said "AttributeError: 'numpy.ndarray' object has no attribute 'save'"

Answer: try using  cv2.imwrite(path,img_to_save)  in the last line.
Question: I started watching a Python course on YouTube in which the guy giving the lesson teaches using VSCode. He started with software installation (Python & Pycharm). Then, in VSCode he downloaded the Python extension (the one made by Microsoft) and the extension called "Code Runner" to run the Python code on VSCode. When I try running my code it hits me with the following error which you can also see in the image on the link at the end of the question. I'm not able to post a screenshot of it because I'm new on this platform. Thanks to whoever sees this. [Running] python -u "c:\Users\Ryan\Desktop\Python\app.py" Python was not found but can be installed from the Microsoft Store: htttps://go.microsoft.com/fwlink?linkID=2082640 Screenshot of the VSCode error screen: 

Answer: You don't have the command python installed into your PATH on Windows which is the default if you didn't get your copy of Python from the Windows Store. If you selected your Python interpreter in VS Code (look in the status bar), then I would disable Code Runner. That way the Python extension is what provides the ability to run Python (the Play button will be green instead of white).