eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I understand that numpy hstack stacks column wise and vstack stacks row wise. Then what is the function of numpy stack?

Answer: The key difference is in the documentation for np.stack (emphasis mine):   Join a sequence of arrays along a new axis.  Consider the following arrays: arr1=np.array([[1,2,3],[7,8,9]]) arr2=np.array([[4,5,6],[10,11,12]]) arr3=np.array([['a','b','c'],['d','e','f']]) [[1 2 3] [7 8 9]] [[ 4 5 6] [10 11 12]] [['a' 'b' 'c'] ['d' 'e' 'f']]  Then consider the following results: #stack horizontally on existing axis np.hstack([arr1,arr2,arr3]) array([['1', '2', '3', '4', '5', '6', 'a', 'b', 'c'], ['7', '8', '9', '10', '11', '12', 'd', 'e', 'f']], dtype='<U11') shape: (2, 9) #stack vertically on existing axis np.vstack([arr1,arr2,arr3]) array([['1', '2', '3'], ['7', '8', '9'], ['4', '5', '6'], ['10', '11', '12'], ['a', 'b', 'c'], ['d', 'e', 'f']], dtype='<U11') shape: (6, 3) #stack depth-wise, adding an axis 2 np.dstack([arr1,arr2,arr3]) array([[['1', '4', 'a'], ['2', '5', 'b'], ['3', '6', 'c']], [['7', '10', 'd'], ['8', '11', 'e'], ['9', '12', 'f']]], dtype='<U11') shape: (2, 3, 3)  Note that in all cases but dstack, the two arrays are joined along an existing axis (axis 0, axis 1, and dstack adds a new axis 2) Then, in light of the above results, what if we use stack instead, only changing the stacking axis? for i in [0,1,2]: stacked=np.stack([arr1,arr2,arr3],axis=i) print(f'Stacked on axis {i}\n',stacked, '\n',f'array shape:{stacked.shape}','\n') Stacked on axis 0 [[['1' '2' '3'] ['7' '8' '9']] [['4' '5' '6'] ['10' '11' '12']] [['a' 'b' 'c'] ['d' 'e' 'f']]] array shape:(3, 2, 3) Stacked on axis 1 [[['1' '2' '3'] ['4' '5' '6'] ['a' 'b' 'c']] [['7' '8' '9'] ['10' '11' '12'] ['d' 'e' 'f']]] array shape:(2, 3, 3) Stacked on axis 2 [[['1' '4' 'a'] ['2' '5' 'b'] ['3' '6' 'c']] [['7' '10' 'd'] ['8' '11' 'e'] ['9' '12' 'f']]] array shape:(2, 3, 3)  Note that these are all 3 dimensional arrays, only the order of the elements changes based on the direction they are stacked
Question: I am trying to setup an AWS lambda that will start an SSM session in my EC2 instance and run a command. For simplicity right now I am just trying to run ls. Here is my lambda function: import json import boto3 def lambda_handler(commands, context): """Runs commands on remote linux instances :param client: a boto/boto3 ssm client :param commands: string, each one a command to execute on the instance :return: the response from the send_command function (check the boto3 docs for ssm client.send_command() ) """ client = boto3.client('ssm') print ("Hello world") resp = client.send_command( DocumentName="AWS-RunShellScript", # Would normally pass commands param here but hardcoding it instead for testing Parameters={"commands":["ls"]}, InstanceIds=["i-01112223333"], ) return resp  However when I run "Test" on this function I get the following log output: START RequestId: d028de04-4004-4a0p-c8d2-975755c84777 Version: $LATEST Hello world [ERROR] Runtime.MarshalError: Unable to marshal response: datetime.datetime(2020, 5, 12, 19, 34, 36, 349000, tzinfo=tzlocal()) is not JSON serializable END RequestId: d028de04-4asdd4-4a0f-b8d2-9asd847813 REPORT RequestId: d42asd04-44d4-4a0f-b9d2-275755c6557 Duration: 1447.76 ms Billed Duration: 1500 ms Memory Size: 128 MB Max Memory Used: 76 MB Init Duration: 301.68 ms  I am not sure what this Runtime.MarshalError: Unable to marshal response error means or how to fix it. The payload I'm passing to run the Test lambda shouldn't matter since I'm not using the commands parameter but it is: { "command": "ls" }  Any help would be appreciated

Answer: The error has pretty clear wording, you're just fixating on wrong part. I assume you use the resp object somewhere down the line and that part tries to do something like json.load() or related. datetime.datetime(2020, 5, 12, 19, 34, 36, 349000, tzinfo=tzlocal()) is not JSON serializable  This is a common error for people running into datetime for the first time, here's pretty comprehensive question about it: How to overcome "datetime.datetime not JSON serializable"?
Question: I want to create a column with the maximum value between 2 values calculated from other columns of the data frame.  import pandas as pd df = pd.DataFrame({"A": [1,2,3], "B": [-2, 8, 1]}) df['Max Col'] = max(df['A']*3, df['B']+df['A']) ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().  The desired outcome is a new df column ['Max Col'] with the maximum value of the above calculations.  I know there is the long solution of creating two new columns with the calculations and then apply .max(axis=1). I am looking for a straight solution.  Thanks.

Answer: Use np.maximum: df['max'] =np.maximum(df['A']*3, df['B']+df['A'])  Output:  A B max 0 1 -2 3 1 2 8 10 2 3 1 9
Question: How to use plotly.graph_objs to plot pandas data in a similar way to plotly.express - specifically to color various data types? The plotly express functionality to group data types based on a value in a pandas column is really useful. Unfortunately I can't use express in my system (as I need to send the graph object to orca) I'm able to get the same functionality by specifically mapping Type to colours (full_plot in the example below), however I have types A-Z, is there a better way of mapping each possible Type in the dataframe to a color? import pandas as pd import plotly.express as px import plotly.graph_objs as go d = {'Scenario': [1, 2, 3, 1, 2,3], 'Type': ["A", "A", "A", "B", "B", "B"], 'VAL_1': [100, 200, 300, 400 , 500, 600], 'VAL_2': [1000, 2000, 3000, 4000, 5000, 6000]} df = pd.DataFrame(data=d) def quick_plot(df): fig = px.bar(df, y='VAL_1', x='Scenario', color="Type", barmode='group') fig['layout'].update(title = "PX Plot", width = 600, height = 400, xaxis = dict(showgrid=False)) fig.show() def full_plot(df): colors = {'A': 'blue', 'B': 'red'} s0=df.query('Type=="A"') s1=df.query('Type=="B"') fig = go.Figure() fig.add_trace(go.Bar( name='A', y=s0['VAL_1'],x=s0['Scenario'], marker={'color': colors['A']})) fig.add_trace(go.Bar( name='B', y=s1['VAL_1'],x=s1['Scenario'], marker={'color': colors['B']})) fig['layout'].update(title = "Full Plot", width = 600, height = 400) fig.update_layout(barmode='group') fig.show() quick_plot(df) full_plot(df) 

Answer: You could simply use a dictionary like this: colors = {'A':'steelblue', 'B':'firebrick'}  The only challenge lies in grouping the dataframe for each unique type and adding a new trace for each type using a for loop. The code snippet below takes care of that to produce this plot:  # imports import pandas as pd import plotly.express as px import plotly.graph_objs as go # data d = {'Scenario': [1, 2, 3, 1, 2,3], 'Type': ["A", "A", "A", "B", "B", "B"], 'VAL_1': [100, 200, 300, 400 , 500, 600], 'VAL_2': [1000, 2000, 3000, 4000, 5000, 6000]} df = pd.DataFrame(data=d) # assign colors to type using a dictionary colors = {'A':'steelblue', 'B':'firebrick'} # plotly figure fig=go.Figure() for t in df['Type'].unique(): dfp = df[df['Type']==t] fig.add_traces(go.Bar(x=dfp['Scenario'], y = dfp['VAL_1'], name=t, marker_color=colors[t])) fig.show()
Question: I have a list of dict which is being converted to a dataframe. When I attempt to pass the columns argument the output values are all nan. # This code does not result in desired output l = [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}] pd.DataFrame(l, columns=['c', 'd']) c d 0 NaN NaN 1 NaN NaN  # This code does result in desired output l = [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}] df = pd.DataFrame(l) df.columns = ['c', 'd'] df c d 0 1 2 1 3 4  Why is this happening?

Answer: Because if pass list of dictionaries from keys are created new columns names in DataFrame constructor: l = [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}] print (pd.DataFrame(l)) a b 0 1 2 1 3 4  If pass columns parameter with some values not exist in keys of dictionaries then are filtered columns from dictonaries and for not exist values are created columns with missing values with order like values in list of columns names: #changed order working, because a,b keys at least in one dictionary print (pd.DataFrame(l, columns=['b', 'a'])) b a 0 2 1 1 4 3 #filtered a, d filled missing values - key is not at least in one dictionary print (pd.DataFrame(l, columns=['a', 'd'])) a d 0 1 NaN 1 3 NaN #filtered b, c filled missing values - key is not at least in one dictionary print (pd.DataFrame(l, columns=['c', 'b'])) c b 0 NaN 2 1 NaN 4 #filtered a,b, c, d filled missing values - keys are not at least in one dictionary print (pd.DataFrame(l, columns=['c', 'd','a','b'])) c d a b 0 NaN NaN 1 2 1 NaN NaN 3 4  So if want another columns names you need rename them or set new one like in your second code.
Question: I'm trying to handle a file, and I need to remove extraneous information in the file; notably, I'm trying to remove brackets [] including text inside and between bracket [] [] blocks, Saying that everything between these blocks including them itself but print everything outside it.  Below is my text File with data sample: $ cat smb Hi this is my config file. Please dont delete it [homes] browseable = No comment = Your Home create mode = 0640 csc policy = disable directory mask = 0750 public = No writeable = Yes [proj] browseable = Yes comment = Project directories csc policy = disable path = /proj public = No writeable = Yes [] This last second line. End of the line.  Desired Output: Hi this is my config file. Please dont delete it This last second line. End of the line.  What i have tried based on my understanding and re-search: $ cat test.py with open("smb", "r") as file: for line in file: start = line.find( '[' ) end = line.find( ']' ) if start != -1 and end != -1: result = line[start+1:end] print(result)  Output: $ ./test.py homes proj 

Answer: with one regex import re with open("smb", "r") as f: txt = f.read() txt = re.sub(r'(\n\[)(.*?)(\[]\n)', '', txt, flags=re.DOTALL) print(txt)  regex explanation: (\n\[) find a sequence where there is a linebreak followed by a [ (\[]\n) find a sequence where there are [] followed by a linebreak (.*?) remove everything in the middle of (\n\[) and (\[]\n) re.DOTALL is used to prevent unnecessary backtracking  !!! PANDAS UPDATE !!! The same solution with the same logic can be carried out with pandas import re import pandas as pd # read each line in the file (one raw -> one line) txt = pd.read_csv('smb', sep = '\n', header=None) # join all the line in the file separating them with '\n' txt = '\n'.join(txt[0].to_list()) # apply the regex to clean the text (the same as above) txt = re.sub(r'(\n\[)(.*?)(\[]\n)', '\n', txt, flags=re.DOTALL) print(txt)
Question: I was playing around with mypy and some basic iteration in Python and wrote the below code base: from typing import Iterator from datetime import date, timedelta class DateIterator: def __init__(self, start_date, end_date): self.start_date = start_date self.end_date = end_date self._total_dates = self._get_all_dates() def _get_all_dates(self) -> Iterator[date]: current_day = self.start_date while current_day <= self.end_date: yield current_day current_day += timedelta(days=1) def __len__(self): print("Calling the len function...") return len(self._total_dates) def __getitem__(self, index): print(f"Calling getitem with value of index as {index}") return self._total_dates[index] if __name__ == "__main__": date_iterator = DateIterator(date(2019, 1, 1), date(2019, 1, 15)) for new_date in date_iterator: print(new_date) date_str = ",".join([str(new_date) for new_date in date_iterator]) print(date_str) print(f"Checking the length of the collection {len(date_iterator)}") print(f"Checking if indexing works : {date_iterator[4]}")  Now to also play around with mypy i got the below issues: iterator_test_with_getitem.py:30: error: Cannot assign to a type iterator_test_with_getitem.py:30: error: "DateIterator" has no attribute "__iter__" (not iterable) iterator_test_with_getitem.py:33: error: "DateIterator" has no attribute "__iter__" (not iterable) Found 3 errors in 1 file (checked 1 source file)  Can someone please guide me that if an object is iterable with the addition of __len__ and __getitem__ methods then why is mypy complaining when it has no iter method Also can someone please tell me what the issue with line 30 is . I don't find any logical explanation of that error as well.

Answer: Mypy -- and PEP 484 type checkers in general -- define an iterable to be any class that defines the __iter__ method. You can see the exact definition of the Iterable type in Typeshed, the collection of type hints for the standard library: https://github.com/python/typeshed/blob/master/stdlib/3/typing.pyi#L146 The reason why Iterable is defined in this way and excludes types that only define __getitem__ and __len__ is because:  The only way to say that a type must either implement __iter__ or implement __getitem__/__len__ is to use a Union -- and defining Iterable to be a union complicates life a bit for anybody who wants to make extensive use of the Iterable type in their own code. Conversely, it's trivial for a class that defines __getitem__ and __len__ to define their own __iter__ method. For example, you could do something as simple as this: def __iter__(self) -> Iterator[date]: for i in range(len(self)): yield self[i]  Or alternatively, something like this (assuming you fixed your constructor so self._total_dates is a list, not a generator): def __iter__(self) -> Iterator[date]: return iter(self._total_dates)   So, given this cost-benefit tradeoff, it makes sense to define Iterable to just be any object that implements __iter__. It's not much of a burden for people who are defining custom classes and simplifies life for people who want to write functions manipulating iterables.
Question: I am analysing extreme weather events. My Dataframe is called df and looks like this: | Date | Qm | |------------|--------------| | 1993-01-01 | 4881.977061 | | 1993-02-01 | 4024.396839 | | 1993-03-01 | 3833.664650 | | 1993-04-01 | 4981.192526 | | 1993-05-01 | 6286.879798 | | 1993-06-01 | 6939.726070 | | 1993-07-01 | 6492.936065 | | ... | ... |  I want to know whether the extreme events happened in the same year as an outlier measured. Thus, I did my boxplot using seaborn: # Qm boxplot analysis boxplot = sns.boxplot(x=df.index.month,y=df['Qm']) plt.show()   Now, I would like to present within the same figure the years corresponding to the outliers. Hence, label them with their date. I have checked in multiple libraries that include boxplots, but there is no clue on how to label them. PD: I used seaborn in this example, but any library that could help will be highly appreciated Thanks!

Answer: You could iterate through the dataframe and compare each value against the limits for the outliers. Default these limits are 1.5 times the IQR past the low and high quartiles. For each value outside that range, you can plot the year next to it. Feel free to adapt this definition if you would like to display more or less years. Here is some code to illustrate the idea. In the code the two last digits of the year are shown next to the position of the outlier. import matplotlib.pyplot as plt import numpy as np import pandas as pd import seaborn as sns Y = 26 df = pd.DataFrame({'Date': pd.date_range('1993-01-01', periods=12 * Y, freq='M'), 'Qm': np.random.normal(np.tile(5000 + 1000 * np.sin(np.linspace(0, 2 * np.pi, 12)), Y), 1000)}) df.set_index('Date', inplace=True) boxplot = sns.boxplot(x=df.index.month, y=df['Qm']) month_q1 = df.groupby(df.index.month).quantile(0.25)['Qm'].to_numpy() month_q3 = df.groupby(df.index.month).quantile(0.75)['Qm'].to_numpy() outlier_top_lim = month_q3 + 1.5 * (month_q3 - month_q1) outlier_bottom_lim = month_q1 - 1.5 * (month_q3 - month_q1) for row in df.itertuples(): month = row[0].month - 1 val = row.Qm if val > outlier_top_lim[month] or val < outlier_bottom_lim[month]: plt.text(month, val, f' {row[0].year % 100:02d}', ha='left', va='center') plt.xlabel('Month') plt.tight_layout() plt.show()
Question: I want to create a form that allows to send a picture with a description using flask forms. I tried to use this video: https://www.youtube.com/watch?v=Exf8RbgKmhM but I had troubles when launching app.py: ➜ website git:(master) ✗ python3.6 app.py Traceback (most recent call last): File "app.py", line 10, in <module> from flask.ext.uploads import UploadSet, configure_uploads, IMAGES ModuleNotFoundError: No module named 'flask.ext'  I had to replace flask.ext.uploads by flask_uploads but now I get: Traceback (most recent call last): File "app.py", line 10, in <module> from flask_uploads import UploadSet, configure_uploads, IMAGES File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/flask_uploads.py", line 26, in <module> from werkzeug import secure_filename, FileStorage ImportError: cannot import name 'secure_filename'  My imports and config looks like this: from datetime import datetime from flask_sqlalchemy import SQLAlchemy from flask import Flask, session, render_template, url_for, redirect, flash, request from wtforms import Form, fields,TextField, StringField, PasswordField, BooleanField,validators from wtforms.validators import InputRequired, Email, Length, DataRequired from flask_wtf import FlaskForm from flask_uploads import UploadSet, configure_uploads, IMAGES from flask_login import LoginManager, UserMixin, login_user, login_required, logout_user, current_user  I couldn't solve this issue, do you have any idea of what can I do ?

Answer: In flask_uploads.py Change  from werkzeug import secure_filename,FileStorage  to from werkzeug.utils import secure_filename from werkzeug.datastructures import FileStorage
Question: I am wondering to create a function which can crop a video in a certain frame and save it on my disk (OpenCV,moviepy,or something like that) I am specifying my function with parameters as dimension of frame along with source and target name (location) def vid_crop(src,dest,l,t,r,b): # something # goes # here left = 1 #any number (pixels) top = 2 # '''' right = 3 # '''' bottom = 4 # '''' vid_crop('myvideo.mp4','myvideo_edit.mp4',left,top,right,bottom)  Any suggestions and ideas are really helpful

Answer: Ok I think you want this, import numpy as np import cv2 # Open the video cap = cv2.VideoCapture('vid.mp4') # Initialize frame counter cnt = 0 # Some characteristics from the original video w_frame, h_frame = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)), int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) fps, frames = cap.get(cv2.CAP_PROP_FPS), cap.get(cv2.CAP_PROP_FRAME_COUNT) # Here you can define your croping values x,y,h,w = 0,0,100,100 # output fourcc = cv2.VideoWriter_fourcc(*'XVID') out = cv2.VideoWriter('result.avi', fourcc, fps, (w, h)) # Now we start while(cap.isOpened()): ret, frame = cap.read() cnt += 1 # Counting frames # Avoid problems when video finish if ret==True: # Croping the frame crop_frame = frame[y:y+h, x:x+w] # Percentage xx = cnt *100/frames print(int(xx),'%') # Saving from the desired frames #if 15 <= cnt <= 90: # out.write(crop_frame) # I see the answer now. Here you save all the video out.write(crop_frame) # Just to see the video in real time cv2.imshow('frame',frame) cv2.imshow('croped',crop_frame) if cv2.waitKey(1) & 0xFF == ord('q'): break else: break cap.release() out.release() cv2.destroyAllWindows()
Question: Hy am using Python RegEx to show all internet wirless profiles connected to a computer.There is error (TypeError: cannot use a string pattern on a bytes-like object) in my Second last line pls anyone help to identifi my mistake.Thanks My Program import subprocess,re command = "netsh wlan show profile" output = subprocess.check_output(command, shell=True) network_names = re.search("(Profile\s*:\s)(.*)", output) print(network_names.group(0))  ..................................................... ERROR line 8, in <module> return _compile(pattern, flags).search(string) TypeError: cannot use a string pattern on a bytes-like object 

Answer: Python 3 distinguishes "bytes" and "string" types; this is especially important for Unicode strings, where each character may be more than one byte, depending on the character and the encoding. Regular expressions can work on either, but it has to be consistent — searching for bytes within bytes, or strings within strings. Depending on what you need, there are two solutions:  Decode the output variable before searching in it; for instance, with: output_text = output.decode('utf-8') This depends on the encoding that you are using; UTF-8 is the most common these days. The matched group will be a string. Search with bytes by adding a b prefix to the regular expression. A regular expression should also use the r prefix, so it becomes: re.search(br"(Profile\s*:\s)(.*)", output) The matched group will be a bytes object.
Question: I have beening using shuffle option for pytorch dataloader for many times. But I was wondering when this shuffle happens and whether it is performed dynamically during iteration. Take the following code as an example: namesDataset = NamesDataset() namesTrainLoader = DataLoader(namesDataset, batch_size=16, shuffle=True) for batch_data in namesTrainLoader: print(batch_data)  When we define "namesTrainLoader", does that mean the shuffling is finished and the following iteration will be based on a fixed order of data? Will there be any randomness in the for loop after namesTrainLoader was defined? I was trying to replace half of "batch_data" with some special value: for batch_data in namesTrainLoader: batch_data[：8] = special_val pre = model(batch_data)  Let us say there will be infinite number of epoches, will "model" eventually see all the data in "namesTrainLoader"? Or half of the data of "namesTrainLoader" is actually lost to "model"?

Answer: The shuffling happens when the iterator is created. In the case of the for loop, that happens just before the for loop starts. You can create the iterator manually with: # Iterator gets created, the data has been shuffled at this point. data_iterator = iter(namesTrainLoader)  By default the data loader uses torch.utils.data.RandomSampler if you set shuffle=True (without providing your own sampler). Its implementation is very straight forward and you can see where the data is shuffled when the iterator is created by looking at the RandomSampler.__iter__ method: def __iter__(self): n = len(self.data_source) if self.replacement: return iter(torch.randint(high=n, size=(self.num_samples,), dtype=torch.int64).tolist()) return iter(torch.randperm(n).tolist())  The return statement is the important part, where the shuffling takes place. It simply creates a random permutation of the indices. That means you will see your entire dataset every time you fully consume the iterator, just in a different order every time. Therefore there is no data lost (not including cases with drop_last=True) and your model will see all data at every epoch.
Question: I have two lists as follows. mylist1 = [["lemon", 0.1], ["egg", 0.1], ["muffin", 0.3], ["chocolate", 0.5]] mylist2 = [["chocolate", 0.5], ["milk", 0.2], ["carrot", 0.8], ["egg", 0.8]]  I want to get the mean of the common elements in the two lists as follows. myoutput = [["chocolate", 0.5], ["egg", 0.45]]  My current code is as follows for item1 in mylist1: for item2 in mylist2: if item1[0] == item2[0]: print(np.mean([item1[1], item2[1]]))  However, since there are two for loops (O(n^2) complexity) this is very inefficient for very long lists. I am wondering if there is more standard/efficient way of doing this in Python.

Answer: You can do it in O(n) (single pass over each list) by converting 1 to a dict, then per item in the 2nd list access that dict (in O(1)), like this: mylist1 = [["lemon", 0.1], ["egg", 0.1], ["muffin", 0.3], ["chocolate", 0.5]] mylist2 = [["chocolate", 0.5], ["milk", 0.2], ["carrot", 0.8], ["egg", 0.8]] l1_as_dict = dict(mylist1) myoutput = [] for item,price2 in mylist2: if item in l1_as_dict: price1 = l1_as_dict[item] myoutput.append([item, (price1+price2)/2]) print(myoutput)  Output: [['chocolate', 0.5], ['egg', 0.45]]