eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to iterate a given list based on a variable number of iterations stored in another list and a constant number of skips stored in as an integer. Let's say I have 3 things -  l - a list that I need to iterate on (or filter) w - a list that tells me how many items to iterate before taking a break k - an integer that tells me how many elements to skip between each set of iterations.  To rephrase, w tells how many iterations to take, and after each set of iterations, k tells how many elements to skip. So, if w = [4,3,1] and k = 2. Then on a given list (of length 14), I want to iterate the first 4 elements, then skip 2, then next 3 elements, then skip 2, then next 1 element, then skip 2. Another example, #Lets say this is my original list l = [6,2,2,5,2,5,1,7,9,4] w = [2,2,1,1] k = 1  Based on w and k, I want to iterate as - 6 -> Keep # w says keep 2 elements 2 -> Keep 2 -> Skip # k says skip 1 5 -> Keep # w says keep 2 elements 2 -> Keep 5 -> Skip # k says skip 1 1 -> Keep # w says keep 1 element 7 -> Skip # k says skip 1 9 -> Keep # w says keep 1 element 4 -> Skip # k says skip 1  I tried finding something from itertools, numpy, a combination of nested loops, but I just can't seem to wrap my head around how to even iterate over this. Apologies for not providing any attempt, but I don't know where to start. I dont necessarily need a full solution, just a few hints/suggestions would do.

Answer: This works: l = [6,2,2,5,2,5,1,7,9,4] w = [2,2,1,1] k = 1 def take(xs, runs, skip_size): ixs = iter(xs) for run_size in runs: for _ in range(run_size ): yield next(ixs) for _ in range(skip_size): next(ixs) result = list(take(l, w, k)) print(result)  Result: [6, 2, 5, 2, 1, 9]  The function is what's called a generator, yielding one part of the result at a time, which is why it's combined into a list with list(take(l, w, k)). Inside the function, the list xs that is passed in is wrapped in an iterator, to be able to take one item at a time with next(). runs defines how many items to take and yield, skip_size defines how many items to skip to skip after each 'run'. As a bonus, here's a fun one-liner - if you can figure out why it works, I think you know enough about the problem to move on :) [y for i, y in zip([x for xs in [[1] * aw + [0] * k for aw in w] for x in xs], l) if i]
Question: Searching for this topic and found a solution but doesn't work for me The code I am working on (part of it like that) pd.set_option('max_colwidth', 1000) df = pd.DataFrame(list)  The line of setting the max colwidth: I tried to put this line before the df line and another try after the df line but the output still the same.. Any ideas?

Answer: You can try setting the others parameters also like: pd.set_option('display.max_columns', 1000, 'display.width', 1000, 'display.max_rows',1000)
Question: There are a few ways to get the list of all Unicode characters' names: for example using Python module unicodedata, as explained in List of unicode character names, or using the website: https://unicode.org/charts/charindex.html but here it's incomplete, and you have to open and parse PDF to find the names. But what is the official source / repository of all Unicode character names? (such that if a new character is added, the list is updated, so I'm looking for the initial source for these names, in a machine readable format). I'm looking for a list with just code point and name, in CSV or any other format: code character name ... 0102 LATIN CAPITAL LETTER A WITH BREVE 0103 LATIN SMALL LETTER A WITH BREVE ... 

Answer: The official source for the actual character data (which includes the character names and many, many other details) is the Unicode Character Database. The latest version of the data files can be accessed via http://www.unicode.org/Public/UCD/latest/. Names specifically can be found in the files NamesList.txt. The format of that file is described here. This is the list in CSV format: https://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt
Question: I am aware that you can flush after a print statement by setting flush=True like so: print("Hello World!", flush=True)  However, for cases where you are doing many prints, it is cumbersome to manually set each print to flush=True. Is there a way to set the default to flush=True for Python 3.x? I am thinking of something similar to the print options numpy gives using numpy.set_printoptions.

Answer: You can use partial: from functools import partial print_flushed = partial(print, flush=True) print_flushed("Hello world!")  From the documentation:  The partial() is used for partial function application which “freezes” some portion of a function’s arguments and/or keywords resulting in a new object with a simplified signature.
Question: Description: (simplified)  I have 2 tasks. Within each task I have 3 coroutines. 2 coroutines from the first task fail. (simulated) While processing task results, I am getting one "Task exception was never retrieved" message. I believe this is because exception of only one of the two failed coroutines in that task was processed. How do I process exceptions of both coroutines within task and/or avoid the "Task exception was never retrieved" message?  Code: (simplified) import asyncio async def download(data): filename = "*" if data in ["b", "c"] else data # simulated failure with open(filename, "w") as f: f.write(data) async def coro(data_list): coroutines = [download(data) for data in data_list] for coroutine in asyncio.as_completed(coroutines): await coroutine async def main(): task1 = asyncio.create_task(coro(["a", "b", "c"])) task2 = asyncio.create_task(coro(["d", "e", "f"])) results = await asyncio.gather(task1, task2, return_exceptions=True) for _ in results: pass asyncio.run(main())  Output: (simplified) Task exception was never retrieved future: <Task finished coro=<download() done, defined at D:/myscript.py:2> exception=OSError(22, 'Invalid argument')> Traceback (most recent call last): File "D:/myscript.py", line 4, in download with open(filename, "w") as f: OSError: [Errno 22] Invalid argument: '*' 

Answer: If you want to collect exceptions instead of raising them, you can use asyncio.gather(return_exceptions=True) in coro as well. For example: import asyncio async def download(data): if data in ['b', 'c']: 1/0 # simulate error return 42 # success async def coro(data_list): coroutines = [download(data) for data in data_list] return await asyncio.gather(*coroutines, return_exceptions=True) async def main(): task1 = asyncio.create_task(coro(["a", "b", "c"])) task2 = asyncio.create_task(coro(["d", "e", "f"])) return await asyncio.gather(task1, task2, return_exceptions=True) print(asyncio.run(main()))  This will print: [[42, ZeroDivisionError('division by zero'), ZeroDivisionError('division by zero')], [42, 42, 42]]
Question: I have a data frame called df: Date Sales 01/01/2020 812 02/01/2020 981 03/01/2020 923 04/01/2020 1033 05/01/2020 988 ... ...  How can I get the first occurrence of 7 consecutive days with sales above 1000? This is what I am doing to find the rows where sales is above 1000: In [221]: df.loc[df["sales"] >= 1000] Out [221]: Date Sales 04/01/2020 1033 08/01/2020 1008 09/01/2020 1091 17/01/2020 1080 18/01/2020 1121 19/01/2020 1098 ... ... 

Answer: You can assign a unique identifier per consecutive days, group by them, and return the first value per group (with a previous filter of values > 1000): df = df.query('Sales > 1000').copy() df['grp_date'] = df.Date.diff().dt.days.fillna(1).ne(1).cumsum() df.groupby('grp_date').head(7).reset_index(drop=True)  where you can change the value of head parameter to the first n rows from consecutive days. Note: you may need to use pd.to_datetime(df.Date, format='%d/%m/%Y') to convert dates from strings to pandas datetime, and sort them.
Question: I have a Pandas dataset named titanic I am plotting a bar chart as described in the Seaborn official documentation, using the following code: import seaborn as sns titanic = sns.load_dataset("titanic") sns.catplot(x="sex", y="survived", hue="class", kind="bar", data=titanic)  This produces the following plot:  As you can see, the hue is represented by the class. How can I manually choose the hue order so that I can reverse the current one?

Answer: In order to manually select the hue order of a Seaborn plot, you have to define the desired order as a list and then pass it to the plot function as the argument hue_order . The following code would work: import seaborn as sns titanic = sns.load_dataset("titanic") hue_order = ['Third', 'Second', 'First'] sns.catplot(x="sex", y="survived", hue="class", data=titanic, hue_order=hue_order, kind="bar")
Question: I'm having trouble deploying a FastAPI app on cpanel with Passenger

Answer: You might be able to run your FastAPI app using a2wsgi: In your passenger_wsgi.py: from a2wsgi import ASGIMiddleware from main import app # Import your FastAPI app. application = ASGIMiddleware(app)
Question: I was surprised that this assertion fails: x = 42 x = lambda: x assert x() == 42  It seems that x ends up recursively referring to itself, so that x(), x()(), etc. are all functions. What is the rule used to parse this, and where is this documented? By the way (not unexpectedly given the above), the original value of x has no references left after the lambda definition: class X: def __del__(self): print('deleting') x = X() x = lambda: x # 'deleting' is printed here 

Answer: The variable x is created by the first assignment, and rebound with the second assignment. Since the x in the lambda isn't evaluated until the lambda is called, calling it will evaluate to the most recently assigned value. Note that this is not dynamic scoping - if it were dynamic, the following would print "99", but it prints "<function ...": x = 42 x = lambda: x def test(f): x = 99 print(f()) test(x)
Question: I am exploring the use of protocol buffers and would like to use the new Timestamp data type which is in protobuf3. Here is my .proto file: syntax = "proto3"; package shoppingbasket; import "google/protobuf/timestamp.proto"; message TransactionItem { optional string product = 1; optional int32 quantity = 2; optional double price = 3; optional double discount = 4; } message Basket { optional string basket = 1; optional google.protobuf.Timestamp tstamp = 2; optional string customer = 3; optional string store = 4; optional string channel = 5; repeated TransactionItem transactionItems = 6; } message Baskets { repeated Basket baskets = 1; }  After generating python classes from this .proto file I'm attempting to create some objects using the generated classes. Here's the code: import shoppingbasket_pb2 from google.protobuf.timestamp_pb2 import Timestamp baskets = shoppingbasket_pb2.Baskets() basket1 = baskets.baskets.add() basket1.basket = "001" basket1.tstamp = Timestamp().GetCurrentTime()  which fails with error:  AttributeError: Assignment not allowed to composite field "tstamp" in protocol message object.  Can anyone explain to me why this isn't working as I am nonplussed.

Answer: See Timestamp. I think you want: basket1.tstamp.GetCurrentTime()
Question: I'm working inside a very big github repo, say its structure is like project-root ├── project-1 │ ├── subproject-a │ └── subproject-others └── project-2 ├── subproject-b └── subproject-others  There are many projects, each contains many subprojects. I'm just working on one of the subprojects (e.g. subproject-a). When I opened vscode inside the subproject (it's a python subproject), I noticed that it launches many rg commands like below, and my CPU usage goes above 99%. I wonder what these rg commands are about? Are they just searching for stuffs inside the subproject, or the whole git repo, which contains tens of thousands of files? Why do they consume so many resources? How could I avoid that, please?  /Applications/Visual Studio Code.app/Contents/Resources/app/node_modules.asar.unpacked/vscode-ripgrep/bin/rg --files --hidden --case-sensitive -g **/*.go/** -g **/*.go -g !**/.git -g !**/.svn -g !**/.hg -g !**/CVS -g !**/.DS_Store -g !**/.classpath -g !**/.factorypath -g !**/.project -g !**/.settings -g !**/node_modules -g !**/bower_components -g !**/*.code-search --no-ignore-parent --follow --quiet --no-config --no-ignore-global 

Answer: It turns out that there are four symlink folders with over 700k files in them. These folders are usually ignored in /project-root/.gitginore. So rg by default would ignore searching in them. But here because of --no-ignore-parent --follow flags, they are being searched nonetheless. I added these folders to /project-root/project-1/subproject-a/.gitignore again, and now these rg commands don't take so much cpu resource anymore.
Question: I am using the new python syntax to describe what types my methods return e.g.,: def method(unpacked_message: dict) -> dict:  This seems to break the vscode color scheme  Expected colors:  Environment and vs code extensions:  Python 3.6.9 on ubuntu ms-python.python v2020.11.371526539 tht13.python: Python for VS code v0.2.3 magicstack.magicpython: MagicPython v1.1.0  The code runs flawlessly. Am I doing something wrong ?

Answer: Based on the information you provided, I reproduced the problem you described. Reason: The Syntax Highlighting style provided by the extension "Python for VSCode" is different from the extension "Python". Solution: Please disable the extension "Python for VSCode". before:  after:
Question: I am porting a keras model over to torch and I'm having trouble replicating the exact behavior of keras/tensorflow's 'categorical_crossentropy' after a softmax layer. I have some workarounds for this problem, so I'm only interested in understanding what exactly tensorflow calculates when calculating categorical cross entropy. As a toy problem, I set up labels and predicted vectors >>> import tensorflow as tf >>> from tensorflow.keras import backend as K >>> import numpy as np >>> true = np.array([[0.0, 1.0], [1.0, 0.0]]) >>> pred = np.array([[0.0, 1.0], [0.0, 1.0]])  And calculate the Categorical Cross Entropy with: >>> loss = tf.keras.losses.CategoricalCrossentropy() >>> print(loss(pred, true).eval(session=K.get_session())) 8.05904769897461  This differs from the analytical result >>> loss_analytical = -1*K.sum(true*K.log(pred))/pred.shape[0] >>> print(loss_analytical.eval(session=K.get_session())) nan  I dug into the source code for keras/tf's cross entropy (see Softmax Cross Entropy implementation in Tensorflow Github Source Code) and found the c function at https://github.com/tensorflow/tensorflow/blob/c903b4607821a03c36c17b0befa2535c7dd0e066/tensorflow/compiler/tf2xla/kernels/softmax_op.cc line 116. In that function, there is a comment: // sum(-labels * // ((logits - max_logits) - log(sum(exp(logits - max_logits))))) // along classes // (The subtraction broadcasts along the batch dimension.)  And implementing that, I tried: >>> max_logits = K.max(pred, axis=0) >>> max_logits = max_logits >>> xent = K.sum(-true * ((pred - max_logits) - K.log(K.sum(K.exp(pred - max_logits)))))/pred.shape[0] >>> print(xent.eval(session=K.get_session())) 1.3862943611198906  I also tried to print the trace for xent.eval(session=K.get_session()), but the trace is ~95000 lines long. So it begs the question: what exactly is keras/tf doing when calculating 'categorical_crossentropy'? It makes sense that it doesn't return nan, that would cause training issues, but where does 8 come from?

Answer: The problem is that you are using hard 0s and 1s in your predictions. This leads to nan in your calculation since log(0) is undefined (or infinite). What is not really documented is that the Keras cross-entropy automatically "safeguards" against this by clipping the values to be inside the range [eps, 1-eps]. This means that, in your example, Keras gives you a different result because it flat out replaces the predictions by other values. If you replace your predictions by soft values, you should be able to reproduce the results. This makes sense anyway, since your networks will usually return such values via a softmax activation; hard 0/1 only happens in the case of numerical underflow. If you want to check this for yourself, the clipping happens here. This function is eventually called by the CategoricalCrossentropy function. epsilon is defined elsewhere, but it seems to be 0.0000001 -- try your manual calculation with pred = np.clip(pred, 0.0000001, 1-0.0000001) and you should see the result 8.059047875479163.
Question: I have a Pandas Series produced by df.column.value_counts().sort_index(). | N Months | Count | |------|------| | 0 | 15 | | 1 | 9 | | 2 | 78 | | 3 | 151 | | 4 | 412 | | 5 | 181 | | 6 | 543 | | 7 | 175 | | 8 | 409 | | 9 | 594 | | 10 | 137 | | 11 | 202 | | 12 | 170 | | 13 | 446 | | 14 | 29 | | 15 | 39 | | 16 | 44 | | 17 | 253 | | 18 | 17 | | 19 | 34 | | 20 | 18 | | 21 | 37 | | 22 | 147 | | 23 | 12 | | 24 | 31 | | 25 | 15 | | 26 | 117 | | 27 | 8 | | 28 | 38 | | 29 | 23 | | 30 | 198 | | 31 | 29 | | 32 | 122 | | 33 | 50 | | 34 | 60 | | 35 | 357 | | 36 | 329 | | 37 | 457 | | 38 | 609 | | 39 | 4744 | | 40 | 1120 | | 41 | 591 | | 42 | 328 | | 43 | 148 | | 44 | 46 | | 45 | 10 | | 46 | 1 | | 47 | 1 | | 48 | 7 | | 50 | 2 |  my desired output is | bin | Total | |-------|--------| | 0-13 | 3522 | | 14-26 | 793 | | 27-50 | 9278 |  I tried df.column.value_counts(bins=3).sort_index() but got | bin | Total | |---------------------------------|-------| | (-0.051000000000000004, 16.667] | 3634 | | (16.667, 33.333] | 1149 | | (33.333, 50.0] | 8810 |  I can get the correct result with a = df.column.value_counts().sort_index()[:14].sum() b = df.column.value_counts().sort_index()[14:27].sum() c = df.column.value_counts().sort_index()[28:].sum() print(a, b, c) Output: 3522 793 9270  But I am wondering if there is a pandas method that can do what I want. Any advice is very welcome. :-)

Answer: You can use pd.cut: pd.cut(df['N Months'], [0,13, 26, 50], include_lowest=True).value_counts()   Update you should be able to pass custom bin to value_counts: df['N Months'].value_counts(bins = [0,13, 26, 50])  Output: N Months (-0.001, 13.0] 3522 (13.0, 26.0] 793 (26.0, 50.0] 9278 Name: Count, dtype: int64
Question: I've used sqlite3 in Python in which execute() creates ambiguity. When I use: import sqlite3 A = sqlite3.connect('a') A.execute('command to be executed') help(A.execute)  I got the output of help() as: ..... ..... Executes a SQL statement. Non-standard.  But when I execute like this: import sqlite3 A = sqlite.connect('a').cursor() A.execute('command to be executed') help(A.execute)  I got the output of help() as: ..... ..... Executes a SQL statement.  My doubt is what does Non-standard refer to? Even the Python documentation provides these words execute for execute(), executemany(), and executescript() in connection objects. I've even searched in web about nonstandard shortcuts in Python. But I didn't get any relevant information. Can anyone help me with this?

Answer: The "nonstandard" function is the execute method of the sqlite3.Connection class:  This is a nonstandard shortcut that creates a cursor object by calling the cursor() method, calls the cursor’s execute() method with the parameters given, and returns the cursor.  "Standard" refers to PEP 249 -- Python Database API Specification v2.0 which the sqlite3 module follows. It does not specify an execute method for the Connection class, but the sqlite3 module provides it anyway, that's why it is called "nonstandard". PEP 249 only specifies the execute method of the Cursor class, which the sqlite3 module implements, of course.
Question: Pip installation is stuck in an infinite loop if there are unresolvable conflicts in dependencies. To reproduce, pip==20.3.0 and: pip install pyarrow==2.0.0 azureml-defaults==1.18.0 

Answer: Workarounds: Local environment: Downgrade pip to < 20.3 Conda environment created from yaml: This will be seen only if conda-forge is highest priority channel, anaconda channel doesn't have pip 20.3 (as of now). To mitigate the issue please explicitly specify pip<20.3 (!=20.3 or =20.2.4 pin to other version) as a conda dependency in the conda specification file AzureML experimentation: Follow the case above to make sure pinned pip resulted as a conda dependency in the environment object, either from yml file or programmatically
Question: I have seen two ways of acquiring the asyncio Lock: async def main(lock): async with lock: async.sleep(100)  and async def main(lock): with await lock: async.sleep(100)  What is the difference between them?

Answer: The second form with await lock is deprecated since Python 3.7 and is removed in Python 3.9. Running it with Python 3.7 gives this warning:  DeprecationWarning: 'with await lock' is deprecated use 'async with lock' instead  Sources (scroll to the bottom):  https://docs.python.org/3.7/library/asyncio-sync.html https://docs.python.org/3.9/library/asyncio-sync.html
Question: When I trained my pytorch model on GPU device,my python script was killed out of blue.Dives into OS log files , and I find script was killed by OOM killer because my CPU ran out of memory.It’s very strange that I trained my model on GPU device but I ran out of my CPU memory. Snapshot of OOM killer log file  In order to debug this issue,I install python memory profiler. Viewing log file from memory profiler, I find when column wise -= operation occurred, my CPU memory gradually increased until OOM killer killed my program. Snapshot of Python memory profiler  It’s very strange, I try many ways to solve this issue.Finally, I found before assignment operation,I detach Tensor first.Amazingly,it solves this issue.But I don’t understand clearly why it works.Here is my original function code. def GeneralizedNabla(self, image): pad_size = 2 affinity = torch.zeros(image.shape[0], self.window_size**2, self.h, self.w).to(self.device) h = self.h+pad_size w = self.w+pad_size #pad = nn.ZeroPad2d(pad_size) image_pad = self.pad(image) for i in range(0, self.window_size**2): affinity[:, i, :, :] = image[:, :, :].detach() # initialization dy = int(i/5)-2 dx = int(i % 5)-2 h_start = pad_size+dy h_end = h+dy # if 0 <= dy else h+dy w_start = pad_size+dx w_end = w+dx # if 0 <= dx else w+dx affinity[:, i, :, :] -= image_pad[:, h_start:h_end, w_start:w_end].detach() self.Nabla=affinity return  If everyone has any ideas,I will appreciate very much, thank you.

Answer: Previously when you did not use the .detach() on your tensor, you were also accumulating the computation graph as well and as you went on, you kept acumulating more and more until you ended up exuasting your memory to the point it crashed. When you do a detach(), you are effectively getting the data without the previously entangled history thats needed for computing the gradients.
Question: I was trying to use jose library for authentication for one of my flask apps. using the import statement as follows from jose import jwt  But it throws following An error, Traceback (most recent call last): File "F:/XXX_XXX/xxxx-services-web/src/auth.py", line 6, in <module> from jose import jwt File "F:\Users\XXXX_XXXXX\AppData\Local\Programs\Python\Python37\lib\site-packages\jose.py", line 546 print decrypt(deserialize_compact(jwt), {'k':key}, ^ SyntaxError: invalid syntax  Is this library outdated?

Answer: installing python-jose instead of jose fixed my problem. https://pypi.org/project/python-jose/