eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have tried inverting a negative film images color with the bitwise_not() function in python but it has this blue tint. I would like to know how I could develop a negative film image that looks somewhat good. Here's the outcome of what I did. (I just cropped the negative image for a new test I was doing so don't mind that) 

Answer: If you don't use exact maximum and minimum, but 1st and 99th percentile, or something nearby (0.1%?), you'll get some nicer contrast. It'll cut away outliers due to noise, compression, etc. Additionally, you should want to mess with gamma, or scale the values linearly, to achieve white balance. I'll apply a "gray world assumption" and scale each plane so the mean is gray. I'll also mess with gamma, but that's just messing around. And... all of that completely ignores gamma mapping, both of the "negative" and of the outputs. import numpy as np import cv2 as cv import skimage im = cv.imread("negative.png") (bneg,gneg,rneg) = cv.split(im) def stretch(plane): # take 1st and 99th percentile imin = np.percentile(plane, 1) imax = np.percentile(plane, 99) # stretch the image plane = (plane - imin) / (imax - imin) return plane  b = 1 - stretch(bneg) g = 1 - stretch(gneg) r = 1 - stretch(rneg) bgr = cv.merge([b,g,r]) cv.imwrite("positive.png", bgr * 255)   b = 1 - stretch(bneg) g = 1 - stretch(gneg) r = 1 - stretch(rneg) # gray world b *= 0.5 / b.mean() g *= 0.5 / g.mean() r *= 0.5 / r.mean() bgr = cv.merge([b,g,r]) cv.imwrite("positive_grayworld.png", bgr * 255)   b = 1 - np.clip(stretch(bneg), 0, 1) g = 1 - np.clip(stretch(gneg), 0, 1) r = 1 - np.clip(stretch(rneg), 0, 1) # goes in the right direction b = skimage.exposure.adjust_gamma(b, gamma=b.mean()/0.5) g = skimage.exposure.adjust_gamma(g, gamma=g.mean()/0.5) r = skimage.exposure.adjust_gamma(r, gamma=r.mean()/0.5) bgr = cv.merge([b,g,r]) cv.imwrite("positive_gamma.png", bgr * 255)   Here's what happens when gamma is applied to the inverted picture... a reasonably tolerable transfer function results from applying the same factor twice, instead of applying its inverse.   Trying to "undo" the gamma while ignoring that the values were inverted... causes serious distortions:   And the min/max values for contrast stretching also affect the whole thing. A simple photo of a negative simply won't do. It'll include stray light that offsets the black point, at the very least. You need a proper scan of the negative.
Question: I have a data.txt file which I want to convert to a data.json file and print a nice first 2 entries (data.txt contains 3 unique IDs). The data.txt can oublicly found here (this is a sample - original file contains 10000 unique "linkedin_internal_id). I tried the following: with open("data.txt", "r") as f: content = f.read() data = json.dumps(content, indent=3)  This code doesn't print the appropriate JSON format of data.txt (it also includes \\). Also, my jupyter notebook gets stacked because of the large file size, for this, I want to nicely print only the first 2 entries.

Answer: It is called new line delimited json where each line is a valid JSON value and the line separator is '\n', you can read it like this line by line and push it to a list, so later it will be easy for you to iterate/process it further. See: ldjson import json with open("data.txt", "r") as f: contents = f.read() data = [json.loads(item) for item in contents.strip().split('\n')] print(data[0:2])
Question: I am new to python and am trying to understand how it handles copies vs references in respect to list unpacking. I have a simple code snippet and am looking for an explanation as to why it is behaving the way it does. arr = [1, 2, 3, 4] [one, two, three, four] = arr print(id(arr[0]), arr[0]) print(id(one), one) one = 5 print(id(one), one)  The output is: (16274840, 1) (16274840, 1) (16274744, 5)  I am not sure why one is all the sudden moved to a different memory location when I try to modify its contents. I am using python version 2.7.18. This is my first post, so I apologize in advance if I am not adhering to the guidelines. Please let me know if I have violated them. Thank you for all the responses. They have helped me boil down my misunderstanding to this code: var = 1 print(id(var), var) var = 5 print(id(var), var)  With output: (38073752, 1) (38073656, 5)  Asking about lists and unpacking them was completely obfuscatory. This does a great job of explaining: http://web.stanford.edu/class/archive/cs/cs106a/cs106a.1212/handouts/mutation.html

Answer: The id/address is not associated with the variable/name; it's associated with the data that the variable is referring to. The 1 object is, in this instance, at address 16274840, and the 5 object is at address 16274744. one = 5 causes one to now refer to the 5 object which is at location 16274744.  Just to rephrase this in terms of C, I think your question essentially boils down to "why does the following not modify the first element of arr?" (I'm ignoring unpacking since it isn't actually relevant to the question): arr = [1, 2, 3, 4] one = arr[0] one = 5  I would approximate that code to the following C which also does not modify arr: int internedFive = 5; int arr[4] = {1, 2, 3, 4}; int* one = &arr[0]; one = &internedFive; printf("%d", arr[0]); // Prints 1  one originally pointed to the first element of arr, but was reassigned to point to the 5. This reassignment of the pointer has no effect on the data location originally pointed to by one and arr[0].
Question: I have a dataframe like this:     col1 col2 col3 col4 col5 col6 col7 col8 col9 col10     ... ... ... ... ... ... ... ... ... ...   ... ... ... ... ... ... ... ... ... ...     and i want to create an xml like this: <?xml version='1.0' encoding='utf-8'?> <root xmlns:xsi="http://www.example.com" xmlns="http://www.example.com"> <all> <col> <col1>...</col1> <col2>...</col2> <col3>...</col3> <col4>...</col4> <col5>...</col5> <col6>...</col6> <group1> <col7>...</col7> <col8>...</col8> </group1> <group2> <col9>...</col9> <col10>...</col10> </group2> </col> <col> <col1>...</col1> <col2>...</col2> <col3>...</col3> <col4>...</col4> <col5>...</col5> <col6>...</col6> <group1> <col7>...</col7> <col8>...</col8> </group1> <group2> <col9>...</col9> <col10>...</col10> </group2> </col> </all> </root>  my solution is to use stylesheet in to_xml function like this: df.to_xml("example.xml", root_name='all', row_name='col', encoding='utf-8', xml_declaration=True, pretty_print=True, index=False, stylesheet='example.xslt')  but i have no idea how to write example.xslt file and how to set to_xml function to get desired xml. I am looking for suggestions and examples of xslt that might work

Answer: Your setup for the to_xml function seems to be ok. In the code below I'm generating a DataFrame with 20 rows and 10 columns to emulate your example. You will find below an example of a xslt file that might work for the sample you have given and the XML output from that. import pandas as pd import numpy as np np.random.seed(42) df = pd.DataFrame(np.random.randint(0,100,size=(20, 10)), columns=[f'col{i+1}' for i in range(10)]) df.to_xml("example.xml", root_name='all', row_name='col', encoding='utf-8', xml_declaration=True, pretty_print=True, index=False, stylesheet='example.xslt')  File example.xslt <?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> <xsl:template match="/"> <root> <all> <xsl:for-each select="all/col"> <col> <col1><xsl:value-of select="col1"/></col1> <col2><xsl:value-of select="col2"/></col2> <col3><xsl:value-of select="col3"/></col3> <col4><xsl:value-of select="col4"/></col4> <col5><xsl:value-of select="col5"/></col5> <col6><xsl:value-of select="col6"/></col6> <group1> <col7><xsl:value-of select="col7"/></col7> <col8><xsl:value-of select="col8"/></col8> </group1> <group2> <col9><xsl:value-of select="col9"/></col9> <col10><xsl:value-of select="col10"/></col10> </group2> </col> </xsl:for-each> </all> </root> </xsl:template> </xsl:stylesheet>  Output File example.xml <?xml version="1.0"?> <root> <all> <col> <col1>51</col1> <col2>92</col2> <col3>14</col3> <col4>71</col4> <col5>60</col5> <col6>20</col6> <group1> <col7>82</col7> <col8>86</col8> </group1> <group2> <col9>74</col9> <col10>74</col10> </group2> </col> <col> <col1>87</col1> <col2>99</col2> <col3>23</col3> <col4>2</col4> ... ... ...
Question: I'm new to Python and working on a task where I need to update a list of dictionaries from a customer input. I have a list as follows: drinks_info = [{'Pepsi': 2.0}, {'Coke': 2.0}, {'Solo': 2.50}, {'Mt Dew': 3.0}]  If the user inputs: Pepsi: 3.0, Sprite: 2.50 Then the list should update to: [{'Pepsi': 3.0}, {'Coke': 2.0}, {'Solo': 2.50}, {'Mt Dew': 3.0}], {'Sprite': 2.50}] Any help is appreciated. Thanks in advance.

Answer:  We can use a function to update list based on user inputs. This function would ask user for each list item separately.  def update_drinks_info(drinks_info): for drink in drinks_info: for key, value in drink.items(): print(key, value) new_price = input("Enter new price for " + key + ": ") if new_price != "": drink[key] = float(new_price) return drinks_info  This function asks user to enter the new price for the specified item, if user presses Enter, then the price stays same without any additional input.  We can use a function to update list based on user specified keys and values.  def update_drinks_info(drinks_info, drink, price): index = None for i in range(len(drinks_info)): if drink in drinks_info[i]: index = i if index is None: drinks_info.append({drink: price}) else: drinks_info[index] = {drink: price} return drinks_info def split_input(input): drinks = input.split(", ") for drink in drinks: drink = drink.split(":") drink[1] = float(drink[1]) update_drinks_info(drinks_info, drink[0], drink[1])  Using split_input function, we are dividing user input to separate arrays and update the drinks info using the specified drink and price. For example: split_input(input("Enter drink and price: "))  with the example input of Pepsi: 3.0, Sprite: 2.50 is going to update the price of both items.
Question: I did a token check, if at least one token is missing, 'True' will not be. Now I need to deduce which variable is missing, how to do it? PRACTICUM_TOKEN = os.getenv('PRACTICUM_TOKEN') TELEGRAM_TOKEN = os.getenv('TELEGRAM_TOKEN') TELEGRAM_CHAT_ID = os.getenv('TELEGRAM_CHAT_ID') def check_tokens(): """Checks the availability of environment variables.""" ENV_VARS = [PRACTICUM_TOKEN, TELEGRAM_TOKEN, TELEGRAM_CHAT_ID] if not all(ENV_VARS): print('Required environment variables are missing:', ...) else: return True 

Answer: I might suggest putting these values inside a class. The check tokens method can be part of the class, and you can use __dict__ to dynamically get reference to all of the tokens you defined without having to duplicate code. class Environment: def __init__(self): self.PRACTICUM_TOKEN = os.getenv('PRACTICUM_TOKEN') self.TELEGRAM_TOKEN = os.getenv('TELEGRAM_TOKEN') self.TELEGRAM_CHAT_ID = os.getenv('TELEGRAM_CHAT_ID') def check_tokens(self): """Checks the availability of environment variables.""" missing_vars = [var for var, value in self.__dict__.items() if not value] if missing_vars: print('Required environment variables are missing:', *missing_vars) return False else: return True print(Environment().check_tokens())
Question: I have the following Pandas DataFrame:  A B 0 Exporter Invoice No. & Date 1 ABC PVT LTD. ABC/1234/2022-23 DATED 20/08/2022 2 1234/B, XYZ, 3 ABCD, DELHI, INDIA Proforma Invoice No. Date. 4 AB/CDE/FGH/2022-23/1234 20.08.2022 5 Consignee Buyer (If other than consignee) 6 ABC Co. 8 P.O BOX NO. 54321 9 Berlin, Germany  Now I want to search for a value in this DataFrame, and store the index and column name in 2 different variables. For example: If I search "Consignee", I should get index = 5 column = 'A' 

Answer: Assuming you really want the index/column of the match, you can use a mask and stack: df.where(df.eq('Consignee')).stack()  output: 5 A Consignee dtype: object  As list: df.where(df.eq('Consignee')).stack().index.tolist()  output: [(5, 'A')]
Question: here is the pandas code that i did to understand how it works for multiple columns. I thought, it sorts columns independently but it did not work like that. df = pd.DataFrame({ 'col1' : ['A', 'Z', 'E', np.nan, 'D', 'C','B'], 'col2' : [2, 1, 9, 8, 7, 4,10], 'col3': [0, 1, 9, 4, 2, 3,1], 'col4': [11,12,12,13,14,55,56], }) df_sort1= df.sort_values(by=['col1', 'col2','col3']) df_sort2= df.sort_values(by=['col1']) #this also return same result #df.sort_values(by=['col1', 'col2','col3','col4']) #df.sort_values(by=['col1', 'col2'])  output of the df_sort1 and df_sort2 is the same. could someone explain please how it works? and what did I not understand here properly? Thanks in advance.

Answer: df_sort2 will sort the dataframe only on col1 value but df_sort1 will do the sorting considering all three columns, if there is a tie break i.e if two rows have same col1 value then it will check for the value of col2 in case col2 value have same value in both the row then it will look after col3 value Lets take the example: import pandas as pd df = pd.DataFrame({ 'col1' : ['A', 'A', 'E', np.nan, 'D', 'C','B'], 'col2' : [2, 1, 9, 8, 7, 4,10], 'col3': [0, 1, 9, 4, 2, 3,1], 'col4': [11,12,12,13,14,55,56], }) print(df.head()) col1 col2 col3 col4 0 A 2 0 11 1 A 1 1 12 2 E 9 9 12 3 NaN 8 4 13 4 D 7 2 14 df_sort1= df.sort_values(by=['col1', 'col2','col3']) print(df_sort1) col1 col2 col3 col4 1 A 1 1 12 0 A 2 0 11 6 B 10 1 56 5 C 4 3 55 4 D 7 2 14 2 E 9 9 12 3 NaN 8 4 13 df_sort2= df.sort_values(by=['col1']) print(df_sort2) col1 col2 col3 col4 0 A 2 0 11 1 A 1 1 12 6 B 10 1 56 5 C 4 3 55 4 D 7 2 14 2 E 9 9 12 3 NaN 8 4 13
Question: I am running Ubuntu 22.04 with xorg. I need to find a way to compile microbit python code locally to a firmware hex file. Firstly, I followed the guide here https://microbit-micropython.readthedocs.io/en/latest/devguide/flashfirmware.html. After a lot of debugging, I got to this point: https://pastebin.com/MGShD31N However, the file platform.h does exist. sawntoe@uwubuntu:~/Documents/Assignments/2022/TVP/micropython$ ls /home/sawntoe/Documents/Assignments/2022/TVP/micropython/yotta_modules/mbed-classic/api/platform.h /home/sawntoe/Documents/Assignments/2022/TVP/micropython/yotta_modules/mbed-classic/api/platform.h sawntoe@uwubuntu:~/Documents/Assignments/2022/TVP/micropython$  At this point, I gave up on this and tried using Mu editor with the AppImage. However, Mu requires wayland, and I am on xorg. Does anyone have any idea if this is possible? Thanks.

Answer: Okay, so elaborating on Peter Till's answer. Firstly, you can use uflash: uflash path/to/your/code .  Or, you can use microfs: ufs put path/to/main.py
Question: I need to get the substring which is continuous more than one char This is my code: l = [] p = 'abbdccc' for i in range(len(p)-1): m = '' if p[i] == p[i+1]: m +=p[i] l.append(m) print(l)   My string is 'abbdccc' b and c are repeated more than 1 times expected out is ['bb', 'ccc']  if My string is '34456788' then my out is ['44', '88']

Answer: Solution with groupby from itertools import groupby [v for _, g in groupby(s) if (v := ''.join(g)) and len(v) > 1]  Sample run for input string s: # input: 'abbdccc' # output: ['bb', 'ccc'] # input: '34456788' # output: ['44', '88']
Question: I wrote this for python code and got an unexpected output. The output was a number of zeros then it said "restarting kernel". Why is the kernel restarting? def countdown(n): for n in range(0,5): print(n) countdown(n-1) countdown(2)  On the other hand, I tried with if and there was no problem: def countdown(n): if n == 0: print("blast of") else: print(n) countdown(n-1) countdown(5)  So why is it not working with for?

Answer: > def countdown(n): > for n in range(0,5): > print(n) > countdown(n-1) > countdown(2)  In your code above, each function call will call itself recursively 5 times. So the first call is 5, second call there will be 25 calls, third call 125 calls, and the recursive calls went exponentially high, causing the Kernel to restart. If you use recursive function, there must be an exit condition. There are a few ways to achieve your goal: (1) recursive with if-condition exit (this is your successful code) def countdown(n): if n == 0: print("blast off!") else: print(n) countdown(n-1) countdown(5)  (2) recursive with while-condition exit def countdown(n): while n != 0: print(n) countdown(n-1) return print("blast off!") countdown(5)  (3) for loop (no need recursive) def countdown(n): for i in range(n, 0, -1): print(i) print("blast off!") countdown(5)  Output: 5 4 3 2 1 blast off!
Question: Can I write this code in one line? I tried use chain in list comprehension. def divisors(n): result = [] for div in range(1, int(sqrt(n)) + 1): if n % div == 0: result.extend([div, n / div]) return list(set(result)) 

Answer: Are you looking for something like this ? from math import sqrt import itertools def divisors(n): return list(set(itertools.chain.from_iterable([[div, n / div] for div in range(1, int(sqrt(n)) + 1) if n % div == 0])))
Question: I plotted a heatmap using seaborn as shown below. I want to change the highlighted column into "Thousands K and Million M" values (shown in the table below). I tried it doing it but the column is changing into string and giving me an error when I am trying to plot those values on the heatmap. Is there a way I can change the values of first column to the desired values on the heatmap?      Values Desired Values     662183343.70 662.83M   155554910.90 155.55M    Code used for creating the heatmap sns.heatmap(heatmap_df, cmap='rocket_r', annot=True, fmt='.4f', linewidths=2, cbar_kws={'label': 'Percentiles', 'orientation': 'vertical', }, vmin=0.95, vmax=1, xticklabels=xticks_label, )  Changing the column into K, M format # convert zeros to K, M etc. from numerize import numerize as nz df['PropDmgAdj'] = df['PropDmgAdj'].apply(nz.numerize) 

Answer: Using the lesser known texts attribute of matplotlib.Axes object: import pandas as pd import seaborn as sns import matplotlib.pyplot as plt heatmap_df = pd.DataFrame( [ {"colA": 662183343.70, "colB": 0.9976, "colC": 0.9962}, {"colA": 155567736.90, "colB": 1.0000, "colC": 1.0000}, {"colA": 77777.70, "colB": 0.9976, "colC": 0.9962}, {"colA": 14456.20, "colB": 0.1243, "colC": 0.5356}, ] ) heatmap = sns.heatmap( heatmap_df, cmap="rocket_r", annot=True, fmt=".4f", linewidths=2, cbar_kws={"label": "Percentiles", "orientation": "vertical",}, vmin=0.95, vmax=1, ) # https://stackoverflow.com/questions/37602885/adding-units-to-heatmap-annotation-in-seaborn for t in heatmap.texts: current_text = t.get_text() # https://stackoverflow.com/questions/67629794/pandas-format-large-numbers text_transform = ( lambda x: f"{x//1000000000}B" if x / 1000000000 >= 1 else f"{x//1000000}M" if x / 1000000 >= 1 else f"{int(x//1000)}K" if x / 10000 >= 1 else f"{x}" ) t.set_text(text_transform(float(current_text))) plt.show()
Question: Consider df1 = pd.DataFrame("Name":["Adam","Joseph","James","James","Kevin","Kevin","Kevin","Peter","Peter"])  I want to get the index of the unique values in the dataframe. When I do df1["Name"].unique() I get the output as ['Adam','Joseph','James','Kevin','Peter']  But I want to get the location of the first occurrence of each value [0,1,2,4,7] 

Answer: I would suggest to use numpy.unique with the return_index as True. np.unique(df1, return_index=True) Out[13]: (array(['Adam', 'James', 'Joseph', 'Kevin', 'Peter'], dtype=object), array([0, 2, 1, 4, 7], dtype=int64))
Question: I am trying to speed up Python programs using Rust, a language in which I am a total beginner. I wrote a function that counts the occurrences of each possible string of length n within a larger string. For instance, if the main string is "AAAAT" and n=3, the outcome would be a hashmap {"AAA":2,"AAT":1}. I use pyo3 to call the Rust function from Python. The code of the Rust function is: fn count_nmers(seq: &str, n: usize) -> PyResult<HashMap<&str,u64>> { let mut current_pos: usize = 0; let mut counts: HashMap<&str,u64> = HashMap::new(); while current_pos+n <= seq.len() { //print!("{}\n", &seq[current_pos..current_pos+n]); match counts.get(&seq[current_pos..current_pos+n]) { Some(repeats) => counts.insert(&seq[current_pos..current_pos+n],repeats+1), None => counts.insert(&seq[current_pos..current_pos+n],1) }; current_pos +=1; } //print!("{:?}",counts) Ok(counts) }  When I use small values for n (n<10), Rust is about an order of magnitude faster than Python, but as the length of n increases, the gap tends to zero with both functions having the same speed by n=200. (see graph) Times to count for different n-mer lengths (Python black, rust red) I must be doing something wrong with the strings, but I can't find the mistake.  The python code is: def nmer_freq_table(sequence,nmer_length=6): nmer_dict=dict() for nmer in seq_win(sequence,window_size=nmer_length): if str(nmer) in nmer_dict.keys(): nmer_dict[str(nmer)]=nmer_dict[str(nmer)]+1 else: nmer_dict[str(nmer)]=1 return nmer_dict def seq_win(seq,window_size=2): length=len(seq) i=0 while i+window_size <= length: yield seq[i:i+window_size] i+=1  

Answer: You are computing hash function multiple times, this may matter for large n values. Try using entry function instead of manual inserts: while current_pos+n <= seq.len() { let en = counts.entry(&seq[current_pos..current_pos+n]).or_default(); *en += 1; current_pos +=1; }  Complete code here Next, make sure you are running --release compiled code like cargo run --release. And one more thing to take in mind is discussed here, Rust may use non-optimal hash function for your case which you can change. And finally, on large data, most of time is spent in HashMap/dict internals which are not a python, but compiled code. So don't expect it to scale well.
Question: I'm trying to set up the development environment for modifying a Python library. Currently, I have a fork of the library, I cloned it from remote and installed it with pip install -e git+file:///work/projects/dev/git_project@branch#egg=git_project However, it seems that instead of creating a symbolic link with pip install -e to the directory where I cloned my package, pip would copy the package to src/git_project in my virtual environment, making it difficult to modify it from there and push changes to my fork at the same time. Am I missing out on something or pip install -e doesn't actually make a symlink when installing from VCS? I know that I can also do pip install -e git+git:// to install from my remote, but it makes it difficult to see real-time changes I make without pushing my code to this fork all the time. Is there a way I can clone a fork to my local development environment, pip install a specific branch from this cloned repo, and create a symlink link to the actual git_project folder so that I can modify the package there, push changes to my remote, and at the same time import the library anywhere in my environment to see real-time changes I make on my branch without committing anything yet? Thanks for any help!

Answer: pip install -e git+URL means "clone the repository from the URL locally and install". If you already have the repository cloned locally and want to simply install from it: just install without Git: cd /work/projects/dev/git_project git checkout branch pip install -e .