eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: How do I find the indices of the first two elements in a list that are any of the elements in another list? For example: story = ['a', 'b', 'c', 'd', 'b', 'c', 'c'] elementsToCheck = ['a', 'c', 'f', 'h']  In this case, the desired output is a list indices = [0,2] for strings 'a' and 'c'.

Answer: story = ['a', 'b', 'c', 'd', 'b', 'c', 'c'] elementsToCheck = ['a', 'c', 'f', 'h'] out = [] for i, v in enumerate(story): if v in elementsToCheck: out.append(i) if len(out) == 2: break print(out)  Prints: [0, 2]
Question: How do I replace duplicates for each group with NaNs while keeping the rows? I need to keep rows without removing and perhaps keeping the first original value where it shows up first. import pandas as pd from datetime import timedelta df = pd.DataFrame({ 'date': ['2019-01-01 00:00:00','2019-01-01 01:00:00','2019-01-01 02:00:00', '2019-01-01 03:00:00', '2019-09-01 02:00:00','2019-09-01 03:00:00','2019-09-01 04:00:00', '2019-09-01 05:00:00'], 'value': [10,10,10,10,12,12,12,12], 'ID': ['Jackie','Jackie','Jackie','Jackie','Zoop','Zoop','Zoop','Zoop',] }) df['date'] = pd.to_datetime(df['date'], infer_datetime_format=True) date value ID 0 2019-01-01 00:00:00 10 Jackie 1 2019-01-01 01:00:00 10 Jackie 2 2019-01-01 02:00:00 10 Jackie 3 2019-01-01 03:00:00 10 Jackie 4 2019-09-01 02:00:00 12 Zoop 5 2019-09-01 03:00:00 12 Zoop 6 2019-09-01 04:00:00 12 Zoop 7 2019-09-01 05:00:00 12 Zoop  Desired Dataframe: date value ID 0 2019-01-01 00:00:00 10 Jackie 1 2019-01-01 01:00:00 NaN Jackie 2 2019-01-01 02:00:00 NaN Jackie 3 2019-01-01 03:00:00 NaN Jackie 4 2019-09-01 02:00:00 12 Zoop 5 2019-09-01 03:00:00 NaN Zoop 6 2019-09-01 04:00:00 NaN Zoop 7 2019-09-01 05:00:00 NaN Zoop  Edit: Duplicated values should only be dropped on the same date indifferent of the frequency. So if value 10 shows up on twice on Jan-1 and three times on Jan-2, the value 10 should only show up once on Jan-1 and once on Jan-2.

Answer: I assume you check duplicates on columns value and ID and further check on date of column date df.loc[df.assign(d=df.date.dt.date).duplicated(['value','ID', 'd']), 'value'] = np.nan Out[269]: date value ID 0 2019-01-01 00:00:00 10.0 Jackie 1 2019-01-01 01:00:00 NaN Jackie 2 2019-01-01 02:00:00 NaN Jackie 3 2019-01-01 03:00:00 NaN Jackie 4 2019-09-01 02:00:00 12.0 Zoop 5 2019-09-01 03:00:00 NaN Zoop 6 2019-09-01 04:00:00 NaN Zoop 7 2019-09-01 05:00:00 NaN Zoop  As @Trenton suggest, you may use pd.NA to avoid import numpy (Note: as @rafaelc sugguest: here is the link explain detail differences between pd.NA and np.nan https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.0.0.html#experimental-na-scalar-to-denote-missing-values) df.loc[df.assign(d=df.date.dt.date).duplicated(['value','ID', 'd']), 'value'] = pd.NA Out[273]: date value ID 0 2019-01-01 00:00:00 10 Jackie 1 2019-01-01 01:00:00 <NA> Jackie 2 2019-01-01 02:00:00 <NA> Jackie 3 2019-01-01 03:00:00 <NA> Jackie 4 2019-09-01 02:00:00 12 Zoop 5 2019-09-01 03:00:00 <NA> Zoop 6 2019-09-01 04:00:00 <NA> Zoop 7 2019-09-01 05:00:00 <NA> Zoop
Question: I have an array A: import numpy as np A = np.array( [0, 0, 1, 1, 1, 0, 1, 1, 0 ,0, 1, 0] )  The length of consecutive '1s' would be: output: [3, 2, 1]  with the corresponding starting indices: idx = [2, 6, 10]  The original arrays are huge and I prefer a solution with less for-loop. Edit (Run time): import numpy as np import time A = np.array( [0, 0, 1, 1, 1, 0, 1, 1, 0 ,0, 1, 0] ) def LoopVersion(A): l_A = len(A) size = [] idx = [] temp_idx = [] temp_size = [] for i in range(l_A): if A[i] == 1: temp_size.append(1) if not temp_idx: temp_idx = i idx.append(temp_idx) else: size.append( len(temp_size) ) size = [i for i in size if i != 0] temp_size = [] temp_idx = [] return size, idx  Quang's solution: def UniqueVersion(A): _, idx, counts = np.unique(np.cumsum(1-A)*A, return_index=True, return_counts=True) return idx, counts  Jacco's solution: def ConcatVersion(A): A = np.concatenate(([0], A, [0])) # get rid of some edge cases starts = np.argwhere((A[:-1] + A[1:]) == 1).ravel()[::2] ends = np.argwhere((A[:-1] + A[1:]) == 1).ravel()[1::2] len_of_repeats = ends - starts return starts, len_of_repeats  Dan's solution (works with special cases as well): def structure(A): ZA = np.concatenate(([0], A, [0])) indices = np.flatnonzero( ZA[1:] != ZA[:-1] ) counts = indices[1:] - indices[:-1] return indices[::2], counts[::2]  Run time analysis with 10000 elements: np.random.seed(1234) B = np.random.randint(2, size=10000) start = time.time() size, idx = LoopVersion(B) end = time.time() print ( (end - start) ) # 0.32489800453186035 seconds start = time.time() idx, counts = UniqueVersion(B) end = time.time() print ( (end - start) ) # 0.008305072784423828 seconds start = time.time() idx, counts = ConcatVersion(B) end = time.time() print ( (end - start) ) # 0.0009801387786865234 seconds start = time.time() idx, counts = structure(B) end = time.time() print ( (end - start) ) # 0.000347137451171875 seconds 

Answer: Here is a pedestrian try, solving the problem by programming the problem. We prepend and also append a zero to A, getting a vector ZA, then detect the 1 islands, and the 0 islands coming in alternating manner in the ZA by comparing the shifted versions ZA[1:] and ZA[-1]. (In the constructed arrays we take the even places, corresponding to the ones in A.) import numpy as np def structure(A): ZA = np.concatenate(([0], A, [0])) indices = np.flatnonzero( ZA[1:] != ZA[:-1] ) counts = indices[1:] - indices[:-1] return indices[::2], counts[::2]  Some sample runs: In [71]: structure(np.array( [0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0] )) Out[71]: (array([ 2, 6, 10]), array([3, 2, 1])) In [72]: structure(np.array( [1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1] )) Out[72]: (array([ 0, 5, 9, 13, 15]), array([3, 3, 2, 1, 1])) In [73]: structure(np.array( [1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0] )) Out[73]: (array([0, 5, 9]), array([3, 3, 2])) In [74]: structure(np.array( [1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1] )) Out[74]: (array([ 0, 2, 5, 7, 11, 14]), array([1, 2, 1, 3, 2, 3]))
Question: In a Flask application, I use a RotatingFileLogger to log werkzeug access logs to a file like shown in this question: file_handler_access_log = RotatingFileHandler("access.log", backupCount=5, encoding='utf-8') formatter = logging.Formatter('%(asctime)s %(module)s %(levelname)s: %(message)s', datefmt='%Y-%m-%d %H:%M:%S') file_handler_access_log.setFormatter(formatter) werkzeug_logger.addHandler(file_handler_access_log) werkzeug_logger.setLevel(logging.DEBUG)  In the access.log file, the request looks like this: 2020-10-07 09:43:51 _internal INFO: 127.0.0.1 - - [07/Oct/2020 09:43:51] "[37mGET /api/foo HTTP/1.1[0m" 200 -  I want to get rid of the color codes like [37m in the log file. The werkzeug documentation states:  The development server can optionally highlight the request logs in different colors based on the status code. Install Click to enable this feature.  Click is a Flask dependency, so I cannot uninstall it. How can I disable the colored logging?

Answer: OK, so what you are hitting is  if click: color = click.style if code[0] == "1": # 1xx - Informational msg = color(msg, bold=True) ... self.log("info", '"%s" %s %s', msg, code, size)   Source: https://github.com/pallets/werkzeug/blob/ef545f0d0bf28cbad02066b4cb7471bea50a93ee/src/werkzeug/serving.py Not easy to prevent the behavior. The second option is to remove color codes from messages. I would try to use log Filter to update the message, something like import logging import click class RemoveColorFilter(logging.Filter): def filter(self, record): if record and record.msg and isinstance(record.msg, str): record.msg = click.unstyle(record.msg) return True remove_color_filter = RemoveColorFilter() file_handler_access_log.addFilter(remove_color_filter)  The above suggestion was inspired by the following answer https://stackoverflow.com/a/60692906/4183498. I didn't test the proposed solution.
Question: I created a lineplot graph to begin with using the following code: plot = sns.lineplot(data=tips, x="sex", y="tip", ci=50, hue="day", palette="Accent") plot.set_title("Value of Tips Given to Waiters, by Days of the Week and Sex", fontsize=24, pad=30, fontdict={"weight": "bold"}) plot.legend("")  I have realised that its actually a catplot chart that I need so I amended the code to the following: plot = sns.catplot (data=tips, x="day", y="tip", kind='bar', ci=50, hue="sex", palette="Accent") plot.set_title("Value of Tips Given to Waiters, by Days of the Week and Sex", fontsize=24, pad=30, fontdict={"weight": "bold"}) plot.legend("")  However I am getting the following error message with the title: 'AttributeError: 'FacetGrid' object has no attribute 'set_title''. Why is my title not working for the catplot chart?

Answer: When you call catplot, it returns a FacetGrid object, so to change the the title and remove legend, you have to use the legend= option inside the function, and also use plot.fig.suptitle() : import seaborn as sns tips = sns.load_dataset("tips") plot = sns.catplot (data=tips, x="day", y="tip", kind='bar', ci=50, hue="sex", palette="Accent", legend=False) plot.fig.suptitle("Value of Tips Given to Waiters, by Days of the Week and Sex", fontsize=24, fontdict={"weight": "bold"})
Question: I am using VSCode for Python along with the Microsoft for Python extension enabled in VSCode. For Python v3.9.0 I am getting No definition found if I try to seek a function definition.  However, I do not get the error if I use my Conda Virtual environment for Python 3.7.0 What might be the problem?

Answer: When I used the code you provided and disabled the Python extension, I encountered the same problem as you. Since "Go to Definition" is supported by the corresponding language service extension, it is recommended that you check that the current Python extension is available and confirm that the selected python interpreter is also available. In addition, please try to reload VSCode.
Question: Background Suppose I am to implement a simple decorator @notifyme that prints a message when the decorated function is invoked. I would like the decorator to accept one argument to print a customized message; the argument (along with the parentheses surrounding the argument) may be omitted, in which case the default message is printed: @notifyme('Foo is invoked!') def foo(): pass @notifyme # instead of @notifyme() def bar(): pass  To allow the parentheses to be omitted, I have to provide two implementations of @notifyme:  The first implementation allows the user to customize the message, so it accepts a string as argument and returns a decorator: def notifyme_customized(message: str) -> Callable[[Callable], Callable]: def decorator(func: Callable) -> Callable: def decorated_func(*args, **kwargs): print(str) return func(*args, **kwargs) return decorated_func return decorator   The second implementation is a decorator itself and uses the first implementation to print a default message: def notifyme_default(func: Callable) -> Callable: return notifyme_customized('The function is invoked.')(func)    To make the two implementations above use the same name notifyme, I used functools.singledispatch to dynamically dispatch the call to notifyme to one of the two implementations: # This is a complete minimal reproducible example from functools import singledispatch from typing import Callable @singledispatch def notifyme(arg): return NotImplemented @notifyme.register def notifyme_customized(message: str) -> Callable[[Callable], Callable]: def decorator(func: Callable) -> Callable: def decorated_func(*args, **kwargs): print(str) return func(*args, **kwargs) return decorated_func return decorator @notifyme.register def notifyme_default(func: Callable) -> Callable: return notifyme_customized('The function is invoked.')(func)  Problem However, as the code is interpreted by the Python interpreter, it complains that typing.Callable is an invalid type: Traceback (most recent call last): File "demo.py", line 20, in <module> def notifyme_default(func: Callable) -> Callable: File "C:\Program Files\Python38\lib\functools.py", line 860, in register raise TypeError( TypeError: Invalid annotation for 'func'. typing.Callable is not a class.  I have found this issue on Python bug tracker, according to which it seems to be expected behavior since Python 3.7. Does a solution or workaround exist in Python 3.8 I use currently (or Python 3.9 that has been released recently)? Thanks in advance.

Answer: I was unable to use typing.Callable with functools.singledispatch, but I did find a workaround by using a function class reference instead: from functools import singledispatch from typing import Callable function = type(lambda: ()) @singledispatch def notifyme(arg): return NotImplemented @notifyme.register def notifyme_customized(message: str) -> Callable[[Callable], Callable]: def decorator(func: Callable) -> Callable: def decorated_func(*args, **kwargs): print(str) return func(*args, **kwargs) return decorated_func return decorator @notifyme.register def notifyme_default(func: function) -> Callable: return notifyme_customized('The function is invoked.')(func)
Question: I am Getting this error. I am executing code of aws lambda function using python 3.7 to know quicksight dashboard version. Thanks in advance!  errorMessage: "Unable to marshal response: Object of type datetime is not JSON serializable",   errorType : "Runtime.MarshalError"  Code- import boto3 import time import sys client = boto3.client('quicksight') def lambda_handler(event, context): response = client.list_dashboard_versions(AwsAccountId='11111', DashboardId='2222',MaxResults=10) return response 

Answer: I quick fix could be: import boto3 import time import sys import json client = boto3.client('quicksight') def lambda_handler(event, context): response = client.list_dashboard_versions(AwsAccountId='11111', DashboardId='2222',MaxResults=10) return json.dumps(response, default=str)
Question: I have a dataframe df and a column df['table'] such that each item in df['table'] is another dataframe with the same headers/number of columns. I was wondering if there's a way to do a groupby like this: Original dataframe: name table Bob Pandas df1 Joe Pandas df2 Bob Pandas df3 Bob Pandas df4 Emily Pandas df5  After groupby: name table Bob Pandas df containing the appended df1, df3, and df4 Joe Pandas df2 Emily Pandas df5  I found this code snippet to do a groupby and lambda for strings in a dataframe, but haven't been able to figure out how to append entire dataframes in a groupby. df['table'] = df.groupby(['name'])['table'].transform(lambda x : ' '.join(x))  I've also tried df['table'] = df.groupby(['name'])['HTML'].apply(list), but that gives me a df['table'] of all NaN. Thanks for your help!!

Answer:  Given 3 dataframes  import pandas as pd dfa = pd.DataFrame({'a': [1, 2, 3]}) dfb = pd.DataFrame({'a': ['a', 'b', 'c']}) dfc = pd.DataFrame({'a': ['pie', 'steak', 'milk']})   Given another dataframe, with dataframes in the columns  df = pd.DataFrame({'name': ['Bob', 'Joe', 'Bob', 'Bob', 'Emily'], 'table': [dfa, dfa, dfb, dfc, dfb]}) # print the type for the first value in the table column, to confirm it's a dataframe print(type(df.loc[0, 'table'])) [out]: <class 'pandas.core.frame.DataFrame'>   Each group of dataframes, can be combined into a single dataframe, by using .groupby and aggregating a list for each group, and combining the dataframes in the list, with pd.concat  # if there is only one column, or if there are multiple columns of dataframes to aggregate dfg = df.groupby('name').agg(lambda x: pd.concat(list(x)).reset_index(drop=True)) # display(dfg.loc['Bob', 'table']) a 0 1 1 2 2 3 3 a 4 b 5 c 6 pie 7 steak 8 milk # to specify a single column, or specify multiple columns, from many columns dfg = df.groupby('name')[['table']].agg(lambda x: pd.concat(list(x)).reset_index(drop=True))  Not a duplicate  Originally, I had marked this question as a duplicate of How to group dataframe rows into list in pandas groupby, thinking the dataframes could be aggregated into a list, and then combined with pd.concat.  df.groupby('name')['table'].apply(list) df.groupby('name').agg(list) df.groupby('name')['table'].agg(list) df.groupby('name').agg({'table': list}) df.groupby('name').agg(lambda x: list(x))   However, these all result in a StopIteration error, when there are dataframes to aggregate.
Question: I have given the following df df = pd.DataFrame(data = {'day': [1, 1, 1, 2, 2, 3], 'pos': 2*[1, 14, 18], 'value': 2*[1, 2, 3]} df   day pos value 0 1 1 1 1 1 14 2 2 1 18 3 3 2 1 1 4 2 14 2 5 3 18 3  and i want to fill in rows such that every day has every possible value of column 'pos' desired result:  day pos value 0 1 1 1.0 1 1 14 2.0 2 1 18 3.0 3 2 1 1.0 4 2 14 2.0 5 2 18 NaN 6 3 1 NaN 7 3 14 NaN 8 3 18 3.0  Proposition: df.set_index('pos').reindex(pd.Index(3*[1,14,18])).reset_index()  yields: ValueError: cannot reindex from a duplicate axis 

Answer: Let's try pivot then stack: df.pivot('day','pos','value').stack(dropna=False).reset_index(name='value')  Output:  day pos value 0 1 1 1.0 1 1 14 2.0 2 1 18 3.0 3 2 1 1.0 4 2 14 2.0 5 2 18 NaN 6 3 1 NaN 7 3 14 NaN 8 3 18 3.0   Option 2: merge with MultiIndex: df.merge(pd.DataFrame(index=pd.MultiIndex.from_product([df['day'].unique(), df['pos'].unique()])), left_on=['day','pos'], right_index=True, how='outer')  Output:  day pos value 0 1 1 1.0 1 1 14 2.0 2 1 18 3.0 3 2 1 1.0 4 2 14 2.0 5 3 18 3.0 5 2 18 NaN 5 3 1 NaN 5 3 14 NaN
Question: I have some noob level virtual environment questions. I've been using virtual environments a little but still have a few questions. I have created and activated an env which is my main working environment as follows: virtualenv env source /path/to/environment/env/bin/activate  Having activated this, I can now see I am in the environment as I have (env) visible on command line. My first question is, do I need to run the activate command each time I open a terminal session? And therefore each time I turn my laptop on, etc.? Further, I want to create another environment that runs on an earlier version of python for testing purposes. I was intending to do it as follows: virtualenv --python=python2.7 env-py2 source /path/to/new/environment/env-py2/bin/activate  Can these virtualenvs be switched easily? So can I activate env-py2 and then easily jump back to activate env again? Or is there an inbetween step required? Apologies for the very basic questions but I was struggling to find high level information.

Answer: Yes you need to run activate command i.e. source each time you open a terminal session. Switching between two virtual environment is easy. You can run deactivate command and source the other virtual environment.
Question: How to access the top element in heapq without deleting (popping) it python ? I need only to check the element at the top of my heapq without popping it. How can I do that.

Answer: From docs python, under heapq.heappop definition, it says: To access the smallest item without popping it, use heap[0]. It says smallest, because it is a min heap. So the item at the top will be the smallest one. Illustration: import heapq pq = [] heapq.heappush(pq,5) heapq.heappush(pq,3) heapq.heappush(pq,1) heapq.heappush(pq,2) heapq.heappush(pq,4) print("element at top = ",pq[0]) print("check the heapq : ", pq)  Result: element at top = 1 check the heapq : [1, 2, 3, 5, 4]
Question: I heard changing XDG_CACHE_DIR or XDG_DATA_HOME fixes that but I did export XDG_CACHE_DIR=<new path> export XDG_DATA_HOME=<new path>  I've also tried pip cache dir --cache-dir <new path>  and pip cache --cache-dir <new path>  and --cache-dir <new path>  and python --cache-dir <new path>  from https://pip.pypa.io/en/stable/reference/pip/#cmdoption-cache-dir and when I type pip cache dir  It's still in the old location. How do I change the directory of pip cache?

Answer: TL;TR;: do not change XDG_CACHE_HOME globally unless you are sure you really want to do that.  Changing XDG_CACHE_HOME globally, like some people suggested would not only affect pip but also other apps as well. You simply do not want to mess that deep, because it's simply not necessary in most of the cases. So what are your alternatives then? You could be using pip's --cache-dir <dir> command line option instead: pip --cache-dir=<dir> ...  Or you could override value of XDG_CACHE_HOME variable for pip invocation only: XDG_CACHE_HOME=<path> pip ...  which also can be made more permanent by i.e. using shell alias feature: alias pip="XDG_CACHE_HOME=<path> pip"  BUT, but, but... there is not need to touch XDG_CACHE_HOME at all, as pip can have own configuration file, in which you can override all of the defaults to match your needs, including alternative location of cache directory. Moreover, all command line switches have accompanying environment variables that pip checks at runtime, which looks like the cleanest approach for your tweaks. In your particular case, --cache-dir can be provided via PIP_CACHE_DIR env variable. So you can either set it globally: export PIP_CACHE_DIR=<path>  or per invocation: PIP_CACHE_DIR=<path> pip ...  or, you create said pip's configuration file and set it there. See docs for more information about pip config file and variables.
Question: I have a Django form which saves a file to s3 through the django-storages library and works fine. How can I generate and return a pre-signed url so the user can access the file temporarily after it is uploaded ? Is this abstracted by django-storages or do I have to use the boto3 api? I have spent hours going through the Django-storages documentation however it is not very clear how to do this .. form.py class DocumentForm(forms.Form): docfile = forms.FileField( label='Select a file', help_text='max. 42 megabytes' ) name = models.CharField(max_length=20) uploaded_at = models.DateTimeField(auto_now_add=True)  views.py def upload_file(request): if request.method == 'POST': form = DocumentForm(request.POST) if form.is_valid(): form.save() url = ... # generate pre-signed url of uploaded file here return render(request, 'upload_response.html', url) 

Answer: Turns out you do not need to use boto3 to generate a presigned url. Django-storages abstracts the entire process. You can simply access the url attribute on the FileField, like in this example: document_form = DocumentForm.objects.get(pk=1) url = document_form.docfile.url  --- Edit ---- For reference, here is the S3 storage class method that generates the pre-signed url for you https://github.com/jschneier/django-storages/blob/770332b598712da27ecdba75c9e202ad6a1a8722/storages/backends/s3boto3.py#L554