eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a CSV file having column names with line breaks when I read the file with pd.read_csv() it returns the column names like this Violent\ncrime\nrate. how do I replace \n with "_" for all these columns?

Answer: Try: df.columns = [c.replace("\n", "_") for c in df.columns] print(df)
Question: In my models I have this: class Example(Basemodel): price = models.IntegerField(default=0)  and in my admin I have this: @admin.register(Example) class ExampleAdmin(admin.ModelAdmin): list_display = ('price',)  I want the price field to be shown in comma-separated format instead of the typical integer format and I want to do that on the backend side. For example: 333222111 should be 333,222,111 Any can recommend me a solution?  Should be: 

Answer: You can work with a property instead, for example: from django.contrib import admin class Example(Basemodel): price = models.IntegerField(default=0) @property @admin.display(description='price', ordering='price') def price_formatted(self): return f'{self.price:,}' and use that property: @admin.register(Example) class ExampleAdmin(admin.ModelAdmin): list_display = ('price_formatted',)
Question: I have the following list: list1 = ['# Heading', '200: Stop Engine', '', '20: Start Engine', '400: Do xy']  and I want to get: list2 = ['20: Start Engine', '200: Stop Engine', '400: Do xy']  So the empty list item and the ones starting with # should be deleted or ignored and the rest should be sorted by the number. I tried to use split() to extract the numbers and the #: list2 = [i.split() for i in list1]  but then I get a list in a list brings some other problems (I need to convert the content of the list to an int for the sorting which only works if I have a string). The output would be: list2 = ['#', 'Heading', '200:', 'Stop', 'Engine', '', '20:', 'Start', 'Engine', '400:', 'Do', 'xy']  and if I split(':'), I can't delete the #. For the sorting I tried: list2.sort(key = lambda x: x[0])  to sort the items by the number. This only works if I can delete the # and the empty item and convert the string to a int. I hope someone can help me! Thanks in advance!

Answer: Just do all the things you say: Ignore all the items which don't start with a number, then sort by the number before the colon delimiter: def FilterAndSort(items): items = [item for item in items if item and item[0].isdigit()] return sorted(items, key=lambda item:int(item.split(':')[0])) print(FilterAndSort(list1))  Output as requested
Question: I am trying to pair elements of a list based on a condition. If two element have a common i will merge them and do this until no elements can be merged. Currently, my problem is looping through same elements and getting same merged result from different items. I have to check if group has been added before.But as my array is empty in the beginning i could not check if element already in it with axis 1. I tried recursive : Also i am discarding if a group has length less than three. pairs = [[1, 3], [1, 8], [2, 1], [2, 3], [3, 1], [3, 8], [4, 11], [4, 15], [7, 13], [9, 12], [9, 13], [10, 1], [10, 18], [10, 20], ...] def groupG(pairs): groups = [] if len(pairs) > 1: for i,pair in enumerate(pairs): try: if (any(point in pairs[i+1] for point in pair)): group = np.concatenate(( pair,pairs[i+1])) group = np.unique(group) groups.append(group) except IndexError: continue if len(groups) == 0 : groupsFiltered = np.array([row for row in pairs if len(row)>=3]) return groupsFiltered else: return groupG(groups)  expected result is : [[1,2,3,8,10,18,20],[4,11,15],[7,9,12,13]...]  Is there a way to group these pairs with while,do while or recursive?

Answer: Use networkx's connected_components: import networkx as nx pairs = [[1, 3], [1, 8], [2, 1], [2, 3], [3, 1], [3, 8], [4, 11], [4, 15], [7, 13], [9, 12], [9, 13], [10, 1], [10, 18], [10, 20]] out = list(nx.connected_components(nx.from_edgelist(pairs)))  output: [{1, 2, 3, 8, 10, 18, 20}, {4, 11, 15}, {7, 9, 12, 13}]
Question: I'm using click v8.1.3 and I'm trying to create some pytests, but I'm not getting my expected return_value when using click.testing.CliRunner().invoke import click.testing import mycli def test_return_ctx(): @mycli.cli.command() def foo(): return "Potato" runner = click.testing.CliRunner() result = runner.invoke(mycli.cli, ["foo"]) assert result.return_value == "Potato" # this fails. b/c the actual value is None  I tried updating the root command to return some random value as well to see if we get a value there # mycli import click @click.group() def cli(): return "Potato"  But it didn't help. return_value for the Result object is still None Am I misunderstanding how I should return a value from a command? https://click.palletsprojects.com/en/8.1.x/api/#click.testing.Result.return_value

Answer: Click command handlers do not return a value unless you use: standalone_mode=False. You can do that during testing like: result = CliRunner().invoke(foo, standalone_mode=False)  Test code: import click from click.testing import CliRunner def test_return_value(): @click.command() def foo(): return "bar" result = CliRunner().invoke(foo, standalone_mode=False) assert result.return_value == "bar"  Test Results: ============================= test session starts ============================ collecting ... collected 1 item test_code.py::test_return_value PASSED [100%] ============================== 1 passed in 0.03s ==============================
Question: I am writing several functions that handle ordered datasets. Sometime, there is an argument that can be a int or float or a timestamp or anything that supports comparison (larger than / smaller than) and that I can use for trimming data for instance. Is there a way to type-hint such a parameter? The typing module doesn't seem to include this, but is there some other way?

Answer: There is no standard 'comparable' ABC, no, as the rich comparison methods are really very flexible and don't necessarily return booleans. The default built-in types return NotImplemented when applied to a type they can't be compared with, for example, while specialised libraries like SQLAlchemy and numpy use rich comparison methods to return completely different objects. See the documentation for the rich comparison methods for the details. But you should be able to define a a Protocol subclass for specific expectations: from typing import Protocol, TypeVar T = TypeVar("T") class Comparable(Protocol[T]): def __eq__(self: T, other: T) -> bool: ... def __lt__(self: T, other: T) -> bool: ... # ... etc  You may need to tweak the protocol to fit your exact expectations, and / or use a non-generic version that's specific to the types you use (perhaps with @overloaded definitions for specific types). For sorting with the builtin sorted() function, __eq__ and __lt__ suffice.
Question: Hi is there any way to get the exact line number where the exception happen? because i am using a wrapper method and in actual method there are many lines of code and i am getting a very generic exception and not sure where exactly it is happening . Eg code as below, import sys def test(**kwargs): print (kwargs) abc def wraper_test(**kwargs): try: test(**kwargs) except Exception as e: exception_type, exception_object, exception_traceback = sys.exc_info() print(exception_object.tfline_no) wraper_test(hello="test", value="lsdf")  Now in the exception line number what i am getting is for test(**kwargs) and not the exact location where the exception is generated in this case "abc" which is inside the test method. Is there any way to capture the exact line number in exception when we are using wrapper method ?

Answer: Try this: the traceback library allows you to get a longer stack trace with more line numbers (this shows the real error is on line 5). import sys, traceback def test(**kwargs): print (kwargs) abc def wrapper_test(**kwargs): try: test(**kwargs) except Exception as e: exception_type, exception_object, exception_traceback = sys.exc_info() traceback.print_tb(exception_traceback, limit=2, file=sys.stdout) wrapper_test(hello="test", value="lsdf")
Question: I am trying to find all permutations of the items inside of the tuples while in the list of length 2. The order of the tuples in relation to each other does not matter. perm = [(3, 6), (6, 8), (4, 1), (7, 4), (5, 3), (1, 9), (2, 5), (4, 8), (5, 1), (3, 7), (6, 9), (10, 2), (7, 10), (8, 2), (9, 10)]  An example of one permutation of the above list would be: perm = [(6, 3), (6, 8), (4, 1), (7, 4), (5, 3), (1, 9), (2, 5), (4, 8), (5, 1), (3, 7), (6, 9), (10, 2), (7, 10), (8, 2), (9, 10)]  Another example permutation would be: perm = [(6, 3), (8, 6), (1, 4), (4, 7), (3, 5), (9, 1), (5, 2), (8, 4), (1, 5), (7, 3), (9, 6), (2, 10), (10, 7), (2, 8), (10, 9)]  In the end, the length of the list of permutations should be 32768, because each tuple is either swapped or not swapped, and 2^15 = 32768. I do not care about the order of the tuples in relation to each other, only the permutations of the items inside of the tuples. I have tried to use itertools permute, combinations, and product, but I haven't been able to get the desired result.

Answer: You can use product: from itertools import product lst = [(3, 6), (6, 8), (4, 1), (7, 4), (5, 3), (1, 9), (2, 5), (4, 8), (5, 1), (3, 7), (6, 9), (10, 2), (7, 10), (8, 2), (9, 10)] output = product(*([(x, y), (y, x)] for x, y in lst)) output = list(output) # if you want a list, rather than a generator print(len(output)) # 32768 print(output[0]) # ((3, 6), (6, 8), (4, 1), (7, 4), (5, 3), (1, 9), (2, 5), (4, 8), (5, 1), (3, 7), (6, 9), (10, 2), (7, 10), (8, 2), (9, 10)) print(output[-1]) # ((6, 3), (8, 6), (1, 4), (4, 7), (3, 5), (9, 1), (5, 2), (8, 4), (1, 5), (7, 3), (9, 6), (2, 10), (10, 7), (2, 8), (10, 9))  The key is to write something like output = product([(3,6), (6,3)], [(6,8), (8,6)], ..., [(9,10), (10,9)])  in a generic way so that any input list would work, which is done by the generator expression and unpacking (*).
Question: When I run: from IPython.core.display import display, HTML display(HTML("<style>.container { width:100% !important; }</style>"))  I got  /var/folders/6g/6gqq4lhx4jbcl4_tbrsxj3xr0000gq/T/ipykernel_5625/333572366.py:1: DeprecationWarning: Importing display from IPython.core.display is deprecated since IPython 7.14, please import from IPython display from IPython.core.display import display, HTML 

Answer: replace from IPython.core.display import display, HTML  with from IPython.display import display, HTML  source here
Question: How can I retain all capital characters, given that subsequent characters are not lower case? Consider this example: import re test1 = 'ThisIsATestTHISISATestTHISISATEST' re.findall(r'[A-Z]{2}[^a-z]+', test1) # ['THISISAT', 'THISISATEST']  Expectation: This: 'THISISAT', should read: 'THISISA'

Answer: Try (regex101): import re test1 = "ThisIsATestTHISISATestTHISISATEST" print(re.findall(r"[A-Z]{2}[A-Z]*(?![a-z])", test1))  Prints: ['THISISA', 'THISISATEST']
Question: How can I find an element using Playwright using a single locator phrase? My element is: <div class="DClass">Hello</div>  I wish to find the element by its class and text: myElement = self.page.locator('text="Hello",[class="DClass"]')  Why it does not work?

Answer: If you separate the selectors with a , that's an or. You can chain selectors using >>. myElement = self.page.locator('text="Hello" >> [class="DClass"]')
Question: I have a question about how to quickly fillna with a sequence in Python(pandas).I have a dataset like following(the true dataset is longer),     Time Number     t0 NA   t1 NA   t2 NA   t3 0   t4 NA   t5 NA   t6 NA   t7 NA   t8 0   t9 NA     My requirement is to add numbers to N lines before and after non-blank lines, and the sequence range is range(-N,N+1).The interval between any two non-empty rows in the dataset is greater than C(constant), our N will be less than C, so there is no need to consider the coverage problem for the time being. Assuming N=2, the result I need is as follows :     Time Number     t0 NA   t1 -2   t2 -1   t3 0   t4 1   t5 2   t6 -2   t7 -1   t8 0   t9 1     At present, the only way I can think of is to use a loop, but the efficiency is low. Does pandas have such a method to do it quickly?

Answer: There are still some unknowns in your question, like what happens if the intervals overlap. Here I will consider that a further interval overwrites the previous one (you can do the other way around with a change of code, see second part). Using rolling, groupby.cumcount, and a mask: s = df['Number'].notna().shift(-N, fill_value=False) m = s.rolling(2*N+1, min_periods=1).max().astype(bool) df['Number2'] = df.groupby(s.cumsum()).cumcount().sub(N).where(m)  NB. I used a slightly different example to show the overlap. output:  Time Number Number2 0 t0 NaN NaN 1 t1 NaN -2.0 2 t2 NaN -1.0 3 t3 0.0 0.0 4 t4 NaN 1.0 5 t5 NaN -2.0 # here we have an overlap, use latter value 6 t6 NaN -1.0 7 t7 0.0 0.0 8 t8 NaN 1.0 9 t9 NaN 2.0 10 t10 NaN NaN  priority on first group s = df['Number'].notna().shift(N, fill_value=False)[::-1] m = s.rolling(2*N+1, min_periods=1).max().astype(bool) df['Number3'] = df.groupby(s.cumsum()).cumcount(ascending=False).rsub(N).where(m)  output:  Time Number Number2 Number3 0 t0 NaN NaN NaN 1 t1 NaN -2.0 -2.0 2 t2 NaN -1.0 -1.0 3 t3 0.0 0.0 0.0 4 t4 NaN 1.0 1.0 5 t5 NaN -2.0 2.0 # difference in behavior 6 t6 NaN -1.0 -1.0 7 t7 0.0 0.0 0.0 8 t8 NaN 1.0 1.0 9 t9 NaN 2.0 2.0 10 t10 NaN NaN NaN
Question:  i have list [31, 32,33, 1,2,3,4, 11,12,13,14] I need to put into adjacent numbers into one list for i, i+1  Expected out [[1,2,3,4], [11,12,13,14], [31, 32,33]] l = [31, 32,33, 1,2,3,4, 11,12,13,14] l.sort() #sorted the items new_l = [] for i in l: temp_l = [] # temp list before appending to main list if i + 1 in l: # if i + 1 is present append to temp_list temp_l.append(i) new_l.append(temp_l) # temp_l has to append to main list  My out is wrong : [[1], [2], [3], [], [11], [12], [13], [], [31], [32], []]

Answer: You can append an empty sub-list to the output list when the difference between the current number and the last number in the last sub-list in the output list is not 1, and keep appending the current number to the last sub-list of the output list: l = [31, 32,33, 1,2,3,4, 11,12,13,14] l.sort() output = [] for i in l: if not output or i - output[-1][-1] != 1: output.append([]) output[-1].append(i)  output becomes: [[1, 2, 3, 4], [11, 12, 13, 14], [31, 32, 33]]  Demo: https://replit.com/@blhsing/UnimportantValidTelecommunications
Question: class Node: def __init__(self,a,b): self._a=a self._b=b def __lt__(self,other): return self._a<other._a a=Node(1,2) b=Node(0,4) print(a>b)  The code above shows True. class Node: def __init__(self,a,b): self._a=a self._b=b def __lt__(self,other): return self._a<other._a def __eq__(self,other): return self._a==other._a a=Node(1,2) b=Node(0,4) print(a>=b)  The code above shows TypeError: '<=' not supported between instances of 'Node' and 'Node. Why defining only lt makes >(which is gt) operation possible? why defining both lt and eq makes <= impossible?

Answer: The Python docs dictates:  There are no swapped-argument versions of these methods (to be used when the left argument does not support the operation but the right argument does); rather, __lt__() and __gt__() are each other’s reflection, __le__() and __ge__() are each other’s reflection, and __eq__() and __ne__() are their own reflection.  So if the left-hand-side argument doesn't implement a comparison operator while the right-hand-side implements its reflection, that reflection is called instead. This also explains why Python doesn't combine __lt__() and __eq__() into __le__() — it simply isn't considered.
Question: I'm really struggling to get this to print the way I want to. I've read through the documentation on removing index, but it seems like it still shows up. Here is my code: quotes = pd.read_csv("quotes.txt",header = None, index_col = False) quote_to_send = quotes.sample(ignore_index = True) print(quote_to_send)  The text file isn't anything special, looks like this: "When you arise in the morning think of what a privilege it is to be alive, to think, to enjoy, to love..." - Marcus Aurelius "Either you run the day or the day runs you." - Jim Rohn ....  The output of this looks like this:  0 0 You may have to fight a battle more than once ...  How do I get rid of those random 0s?

Answer: The 0 on top is your column name, since you don't have one... The 0 on the left is your index, something that absolutely every dataframe needs. If you really want to see things without those essential pieces, you can use print(df.to_string(header=False, index=False)) When you arise in the morning think of what a privilege it is to be alive, to think, to enjoy, to love... - Marcus Aurelius Either you run the day or the day runs you. - Jim Rohn
Question: I have a URL that points to a local file. 'file:///home/pi/Desktop/music/Radio%20Song.mp3'  I need to somehow convert this into a traditional file path, like the os module employs. '/home/pi/Desktop/music/Radio Song.mp3'  Right now I'm hacking it with the replace() method. path = file.replace('file://', '').replace('%20', ' ')  I've looked at the os module, and it doesn't seem to have support for this. I've searched various ways of phrasing it, and I can't seem to find an answer. Am I just ignorant of the terminology? What's the proper way to do this?

Answer: The following would work: from urllib.request import url2pathname from urllib.parse import urlparse p = urlparse('file:///home/pi/Desktop/music/Radio%20Song.mp3') file_path = url2pathname(p.path) print(file_path)  (thanks to user @MillerTime correctly pointing out that the solution will not remove file:// without the urlparse) Output: /home/pi/Desktop/music/Radio Song.mp3  urllib is a standard library, so no installation required. On a Windows machine, running just url2pathname would give you a valid file path right away, but it would be relative to the working directory of the script - e.g. running it from somewhere on D: drive: D:\home\pi\Desktop\music\Radio Song.mp3
Question: I am working on an ETL pipeline using docker airflow. I want to trigger my pipeline whenever any new file is uploaded to S3 bucket. Is there any S3sensor in airflow that checks any new file in bucket? The S3sensor should ignore the existing files in location and should only trigger when new file is added to S3.

Answer: You have several options to achieve this goal:  The best solution is creating S3 Event Notifications on file creation to send a message to SQS. In Airflow you can create a sensor to check if there are new messages to process them. You can also create a sensor which list the files in S3 bucket, and add them to a state store (DB for ex) with state to_process, the next time it will compares between the files list and the files in the state store to know it there are new files or not, then your dag process the records in the state store which have a state != done, and when it finish the processing, it updates the state to done. You can add other metadata like created_at, processed_at, and other states like error to reprocess them in the next run or send an alert to your team.
Question: The goal is to be able to use 1 script to create different reports based on a filter. I want my Databricks Job Task parameters and Notebook variables to share the same value for filtering purposes. This is how I declared these widgets and stored in a variable:   dbutils.widgets.text(name='field', defaultValue='', label='field') f1= dbutils.widgets.get('field')    

Answer: There are two methods to use with widgets [.text() + .get()]. One to create the widget the first time and one to grab the value from the widget. Here is some sample screen shots from a class I teach.  The .text method creates the widget and sets the value. It only has to be executed once. It can be commented out afterwards. This has to be recreated if you move the code from one Databricks workspace to another. In this example, I have a notebook that reads a csv file and performs a full load of a delta table. The source file, destination path, debug flag, file schema and partition count as passed as parameters.  I am using the adventure works data files.  The parent notebook calls the child notebook 15 times to load and create hive tables for the adventure works schema. The same can be done for report file creation. I hope this explains and shows how to use widgets effectively and call a notebook using the dutils.notebook.run() method. You can translate the .run() method calls to tasks in a databricks job. What I do not like is the interface. There is a lot of clicking and entering vs cut/paste when using a notebook. I guess you can use the JSON tab to the right. To be honest, I usually use ADF for scheduling since most of my clients data is hybrid in nature. Here is a screen shot of a sample job with at task to load the accounts hive table.  Last but not least, the job runs to success.
Question: I have a spreadsheet with data. There I have a name like Roger Smith. I would like to have the user name rsmith. Therefore, the first letter of the first name followed by the family name. How can I do this in Python?

Answer: def make_username(full_name: str) -> str: first_names, last_name = full_name.lower().rsplit(maxsplit=1) return first_names[0] + last_name print(make_username("Roger M. Smith"))  Output: rsmith The use of rsplit is to ensure that in case someone has more than one first name, the last name is still taken properly. I assume that last names will not have spaces in them. Note however that depending on your use case, you may need to perform additional operations to ensure that you don't get duplicates.
Question: I am using tqdm to display various progress bars for my Python console application. For the production deployment of the applications, I use Docker. The progress bars work fine when running a Python application in a terminal. However, when Dockerized and the terminal output is accessed through docker logs the progress bar does not function because as far as I understand it is not an interactive terminal. Although it looks like the progress gets rendered if docker logs is dumped after the progress bar have completed, but not sure if there are some other conditions for this to happen (output buffering?). I would like to modify my tqdm behavior so that  It detects when it is run in non-interactive Dockerised environment Instead of displaying interactive progress bar, it will log completion statements (10% done, X iterations/s) regularly  This way the progress durations and such would be more accessible when running the application in production. What would be the way to attach such a custom behavior to tqdm?

Answer: The package tqdm_loggable is a drop in replacement for tqdm that works well for this use case. To install: pip install tqdm-loggable Then just replace any imports of tqdm (from tqdm import tqdm) with: from tqdm_loggable.auto import tqdm Be sure to set logging level to INFO to see the results in the logs: import logging logging.basicConfig(level=logging.INFO)