eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Let's say I have this: class MyPackage ( dict ) : def __init__ ( self ) : super().__init__() def __setitem__ ( self, key, value ) : raise NotImplementedError( "use set()" ) def __getitem__ ( self, key ) : raise NotImplementedError( "use get()" ) def set ( self, key, value ) : # some magic self[key] = value def get ( self, key ) : # some magic if not key in self.keys() : return "no!" return self[key]  (Here, # some magic is additional code that justifies MyPackage as opposed to 'just a dictionary.') The whole point is, I want to provided a dictionary-like object that forces the use of get() and set() methods and disallows all access via [], i.e. it is not permitted to use a['x']="wow" or print( a['x'] ) However, the minute I call get() or set(), the NotImplementedError is raised. Contrast this with, say, Lua, where you can bypass "overloading" by using "raw" getters and setters. Is there any way I can do this in Python without making MyPackage contain a dictionary (as opposed to being a dictionary)?

Answer: You already have the solution in your code: use super(). This allows you to call methods on the parent class, in this case, dict:  def set ( self, key, value ) : # some magic super().__setitem__(key, value) def get ( self, key ) : # some magic if not key in self.keys() : return "no!" return super().__getitem__(key)
Question: I would like to recovery the rows in a dataframe where, in the same row, differing keys hold equal values. I can display where, for instance, the rows where col2 == col3. I would like to get this code to track across col1 matching across col2, col3 and col4. Then col2 to match across col 3 and col4. Then finally col3 across col4. I have read through this post and I am confused if iteration is the solution to my problem. If so, how can this be done. I can display, for instance, the rows where col2 == col3. # -*- coding: utf-8 -*- import pandas as pd ## writing a dataframe rows = {'col1':['5412','5148','5800','2122','5645','1060','4801','1039'], 'col2':['542','512','541','412','565','562','645','152'], 'col3':['542','3120','3410','2112','5650','5620','4801','152'], 'col4':['5800','2122','5645','2112','412','562','562','645'] } df = pd.DataFrame(rows) print(f'Unsorted dataframe \n\n{df}') ## print the rows where col2 == col3 dft = df[(df['col2'] == df['col3'])] print('\n\nupdate - list row of matching row elements') print(dft) ## print all except the rows where col2 == col3 dft = df.drop(df[(df['col2'] == df['col3'])].index) print('\n\nupdate - Dropping rows of matching row elements') print(dft)  With this I am getting back  col1 col2 col3 col4 0 5412 542 542 5800 7 1039 152 152 645  I would like to get back  col1 col2 col3 col4 0 5412 542 542 5800 3 2122 412 2112 2112 4 5645 565 5650 412 5 1060 562 5620 562 6 4801 645 4801 562 7 1039 152 152 645 

Answer: Use nunique with axis=1 and compare it to the number of columns: import pandas as pd rows = { "col1": ["5412", "5148", "5800", "2122", "5645", "1060", "4801", "1039"], "col2": ["542", "512", "541", "412", "565", "562", "645", "152"], "col3": ["542", "3120", "3410", "2112", "5650", "5620", "4801", "152"], "col4": ["5800", "2122", "5645", "2112", "412", "562", "562", "645"], } df = pd.DataFrame(rows) df = df[df.nunique(axis=1) < len(df.columns)] print(df)  Output:  col1 col2 col3 col4 0 5412 542 542 5800 3 2122 412 2112 2112 5 1060 562 5620 562 6 4801 645 4801 562 7 1039 152 152 645
Question: Hi guys I have two different processes running at the same time. I'm trying to stop the other process which i have named loop_b but I have only managed to stop the current process and the other doesn't stop from multiprocessing import Process import time import sys counter=0 def loop_a(): global counter while True: print("a") time.sleep(1) counter+=1 if counter==10: print("I want to stop the loop_b") sys.exit() #Process(target=loop_b).close() -> Also i tried this but not worked def loop_b(): while True: print("b") time.sleep(1) if __name__ == '__main__': Process(target=loop_a).start() Process(target=loop_b).start() 

Answer: One possible solution could be using multiprocessing.Event synchronization primitive to signal other process that it should exit, e.g.: import sys import time from multiprocessing import Event, Process def loop_a(evt): counter = 0 while True: print("a") time.sleep(1) counter += 1 if counter == 3: print("I want to stop the loop_b") evt.set() # <-- set the flag to be True to signal exit sys.exit() def loop_b(evt): while not evt.is_set(): print("b") time.sleep(1) if __name__ == "__main__": evt = Event() p1 = Process(target=loop_a, args=(evt,)).start() p2 = Process(target=loop_b, args=(evt,)).start()  Prints: a b a b a b I want to stop the loop_b # <-- and the loob_b ends here too
Question: I have two topic models, topics1 and topics2. They were created from very similar but different datasets. As a result, the words representing each topic/cluster as well as the topic numbers will be different for each dataset. A toy example looks like: import pandas as pd topics1 = pd.DataFrame({'topic_num':[1,2,3], 'words':[['red','blue','green'], ['blue','sky','cloud'], ['eat','food','nomnom']] }) topics2 = pd.DataFrame({'topic_num':[1,2,3], 'words':[['blue','sky','airplane'], ['blue','green','yellow'], ['mac','bit','byte']] })  For each topic in topics1, I would like to find the topic in topics2 with the maximum number of matches. In the above example, in topics1 topic_num 1 would match topic_num 2 intopics2 and topic_num 2 in topics1 would match topic_num 1 in topics2. In both of these cases, 2 of the 3 words in each row match across dataframes. Is there a way to find this using built-in pandas functions such as eq()? My solution just iterated across every word in topics1 and eery word in topics2.

Answer: You will have to compute all combinations in any case. You could use sets and broadcast the comparisons: # set the topic_num as index # convert the Series of lists to Series of sets s1 = topics1.set_index('topic_num')['words'].map(set) s2 = topics2.set_index('topic_num')['words'].map(set) # convert to numpy arrays # broadcast the intersection of s1 and s2 # to form a square array, convert back to DataFrame # get the length of the intersection tmp = (pd.DataFrame(s1.to_numpy()[:, None] &s2.to_numpy(), index=s1.index, columns=s2.index ) .map(len) #.applymap(len) ) # get the col id with the max length per row # mask if the length was 0 out = tmp.idxmax(axis=1).where(tmp.gt(0).any(axis=1))  Output: topic_num 1 2.0 2 1.0 3 NaN dtype: float64  Intermediate tmp: topic_num 1 2 3 topic_num 1 1 2 0 2 2 1 0 3 0 0 0  Note that this logic can be wrapped by cdist as rightly suggested by @Onyambu, however I would first convert to set to avoid doing it repeatedly in the cdist (which would be expensive). This should be done in a bit weird way by encapsulating the sets in list since cdist requires a 2D input: from scipy.spatial.distance import cdist dst = lambda x, y : len(x[0] & y[0]) val = cdist([[set(x)] for x in topics1['words']], [[set(x)] for x in topics2['words']], dst) pos = val.argmax(1) out = pd.DataFrame({'topic1': topics1['topic_num'], 'topic2': (topics2['topic_num'] .iloc[pos] .where(val[topics1.index, pos]!=0) .values )}) # topic1 topic2 # 0 1 2.0 # 1 2 1.0 # 2 3 NaN
Question: PHP code // php 8.1.13 $text = '1'; $key = base64_decode('3pKtqxNOolyBoJouXWwVYw=='); $iv = base64_decode('O99EDNAif90='); $encrypted = openssl_encrypt($text, 'des-ede3-cbc', $key, OPENSSL_RAW_DATA, $iv); $hex = strtoupper(bin2hex($encrypted)); echo $hex;  The result is FF72E6D454B84A7C Python3.8 pycryptodome 3.20.0 import binascii import base64 from Crypto.Cipher import DES3 from Crypto.Util.Padding import pad key = base64.b64decode('3pKtqxNOolyBoJouXWwVYw==') iv = base64.b64decode('O99EDNAif90=') cipher = DES3.new(key, DES3.MODE_CBC, iv) data = '1' data = data.encode('utf-8') data = pad(data, DES3.block_size) encrypted = cipher.encrypt(data) print(binascii.hexlify(encrypted).decode('utf-8').upper())  The result is A311743FB5D91569 Why are these two results different, and how can I make python's results consistent with PHP's? I've tried using pycryptodome and pyDes and can't achieve the same result as PHP

Answer: The reason for the different results is that different TripleDES variants are used in both codes. des-ede3-cbc in the PHP code specifies 3TDEA, which requires a 24 bytes key. Since the key used is only 16 bytes in size, PHP/OpenSSL silently pads it with 0x00 values to the required size of 24 bytes. With PyCryptodome, the key length determines the variant. Since the key is 16 bytes in size, 2TDEA is applied. For this reason the result is different. In order to also use 3TDEA, the key must be explicitly padded in the Python code, i.e. expanded to 24 bytes with 0x00 values: key = base64.b64decode('3pKtqxNOolyBoJouXWwVYw==') + b'\x00'*8  If this key is applied in the Python code, the result is the same as in the PHP code.
Question: E.g. DF which contains number of executions across timestamps.  DateTime Execution 0 2023-04-03 07:00:00 11 1 2023-04-03 11:00:00 1 2 2023-04-03 12:00:00 1 3 2023-04-03 14:00:00 3 4 2023-04-03 18:00:00 1 <class 'pandas.core.frame.DataFrame'> RangeIndex: 5080 entries, 0 to 5079  Below is the output I'm trying to achieve  DateTime Execution 0 2023-04-03 07:00:00 4 1 2023-04-03 08:00:00 4 2 2023-04-03 09:00:00 3 3 2023-04-03 11:00:00 1 4 2023-04-03 12:00:00 1 5 2023-04-03 14:00:00 3 6 2023-04-03 18:00:00 1  Only if the execution is more than 4, it should be distributed to the next hours. Maximum for any hour is 4. Thanks again for quick help. How to distribute pandas dataframe rows evenly across timestamps based on value of the column This helps with Evenly distribution, I'm looking at uneven distribution.

Answer: With asfreq/clip : N, C = 4, "Execution" asfreq = df.set_index("DateTime").asfreq("h") out = ( (gby:=asfreq.groupby(asfreq[C].notna().cumsum()))[C] .transform("first") .sub(gby.cumcount() * N) .clip(upper=N) .loc[lambda s: s.gt(0)] .reset_index(name=C) .convert_dtypes() )  Output :  DateTime Execution 0 2023-04-03 07:00:00 4 1 2023-04-03 08:00:00 4 2 2023-04-03 09:00:00 3 3 2023-04-03 11:00:00 1 4 2023-04-03 12:00:00 1 5 2023-04-03 14:00:00 3 6 2023-04-03 18:00:00 1
Question: I trained a TensorFlow model for text classification, but I can't use a Polars DataFrame to make my predictions on it. However, I can use a Pandas DataFrame. import pandas as pd import polars as pl import joblib from tensorflow.keras.models import load_model loaded_model =load_model('model.keras') load_Le = joblib.load('label_encoder.joblib')  If i do: text = "some example text" df = pd.DataFrame({"Coment":[text]}) preddict = loaded_model.predict(df["Coment"])  I have no problems, but if I do: text = "some example text" df = pl.DataFrame({"Coment":[text]}) preddict = loaded_model.predict(df["Coment"])  I get TypeError: cannot convert the argument type_value: String to a TensorFlow Dtype. Any advice? Some extra info: Before saving my model, I added this so I can predict on any text (Works fine on pandas) inputs = keras.Input(shape=(1,), dtype="string") processed_inputs = text_vectorization(inputs) outputs = model(processed_inputs) inference_model = keras.Model(inputs, outputs) inference_model.save('model.keras') 

Answer: As of late 2023, there were no plans to make Polars dataframes compatible with TensorFlow. In the meantime you can convert polar frames to pandas with the to_pandas() member and pass those to the models.
Question: I'm struggling a bit trying to create a lookup table of items that are bought together. I have a solution in "brute force" Python iterating over a DataFrame, but I'd prefer one in pure Pandas using some kind of aggregate trick if possible. Here is what I have. I've made a small example with a list of invoice numbers and what item that was bought. import pandas as pd df = pd.DataFrame({ "Invoice": [1, 1, 2, 3, 4, 4, 4, 5, 5], "Item": ["Apple", "Pear", "Banana", "Apple", "Apple", "Orange", "Pear", "Apple", "Orange"] }) df  We can see that invoice number 4 has three items and invoice number 2 has just one. And here is my solution that I'm not really happy with: item_keys = [] item_values = [] for i, row in df.iterrows(): invoice = row["Invoice"] item = row["Item"] for item_2 in df[df["Invoice"] == invoice]["Item"]: if item_2 != item: item_keys.append(item) item_values.append(item_2) lookup_table = pd.DataFrame({"key": item_keys, "value": item_values}) lookup_table  In lookup_table I can then search for a key item and gett all items that were sold together with it. Is there a faster and more elegant way to do this?)

Answer: You can also do a self join and filter down: import pandas as pd df = pd.DataFrame({ "Invoice": [1, 1, 2, 3, 4, 4, 4, 5, 5], "Item": ["Apple", "Pear", "Banana", "Apple", "Apple", "Orange", "Pear", "Apple", "Orange"] }) lookup_table = ( df.merge(df, on='Invoice') .rename(columns={'Item_x': 'key', 'Item_y': 'value'}) .loc[lambda d: (d['key'] != d['value'])] .reset_index(drop=True) # .drop(columns='Invoice') # uncomment to remove the 'Invoice' column ) print(lookup_table) # Invoice key value # 0 1 Apple Pear # 1 1 Pear Apple # 2 4 Apple Orange # 3 4 Apple Pear # 4 4 Orange Apple # 5 4 Orange Pear # 6 4 Pear Apple # 7 4 Pear Orange # 8 5 Apple Orange # 9 5 Orange Apple
Question: import os command = "conda run -n python3.5 python generate_handwriting.py -text '{}' -style {} -bias {} -stroke_color '{}' -stroke_width {} -output '{}'".format(text, style, bias, stroke_color, stroke_width, output_filename) os.system(command)  I get the variables, for example text, directly from the user. I've been notified that this could allow the user to execute malicious code. I have rewritten the code to this: import subprocess cmd = ["conda", "run", "-n", "python3.5", "python", "generate_handwriting.py"] args = ["-text", str(text), "-style", str(style), "-bias", str(bias), "-stroke_color", str(stroke_color), "-stroke_width", str(stroke_width), "-output", output_filename] process = subprocess.run(cmd + args, check=True)  Is the issue fixed now? The full code can be found here.

Answer: Shell Injection The subprocess.run command, as opposed to os.system, doesn't allow for arbitrary shell execution, like redirections or multiple commands in one string, by default. You'd have to explicitely set shell=True for that. Also, you're putting the arguments in a list, which is pretty much the Python equivalent of SQL's prepared statements. Therefore, it is much better to do it like this. That being said, I'd still advise you to escape user input with shlex.quote, just on general principle. For example, imagine you were to pass a single string command to run, instead of a command list, then a malicious user could pass something like -c "my arbitrary command". This shouldn't be an issue in your specific case, but better safe than sorry. Other Issues If this process is running with higher permissions than the user, you'd need to make sure to close other avenues of attack as well. Some obvious ones would be:  Checking that the user has permissions to edit the output destination. Restricting write access to generate_handwriting.py. PATH poisoning, such that the conda executable is provided by the user.  But these are from the top of my head and there might very well be more.
Question: I'm creating a button in StackLayout in file info_options.kv. Now I want to call a function in class InfoOptionsPanel whenever this button is pressed. I'm able to call this funtion when I'm replacing return MainPanel() with return InfoOptionsPanel() i.e. when I'm not using the layout created by MainPanel, but when I'm using MainPanel class for gui then button gives below, also note that there is no problem in GUI even when using MainPanel for GUI. error thrown File "C:\Users\..\AppData\Local\Programs\Python\Python312\Lib\site-packages\kivy\lang\builder.py", line 60, in custom_callback exec(__kvlang__.co_value, idmap) File "K:\path\to\project\Info_options.kv", line 5, in <module> on_press: self.parent.on_press() ^^^^^^^^^^^^^^^^ AttributeError: 'InfoOptionsPanel' object has no attribute 'on_press'  info.kv <InfoPanel@BoxLayout> orientation: 'vertical' size_hint_y: 0.8 pos_hint: {"top": 1} Label: id: info_label text:"Click button" font_size:50  info_options.kv <InfoOptionsPanel@StackLayout>: orientation: 'lr-tb' Button: text: 'Button' on_press: self.parent.on_press() size_hint :.7, 0.1  main_screen.kv <MainPanel@BoxLayout>: orientation: 'vertical' AnchorLayout: anchor_x: 'left' anchor_y: 'top' InfoPanel: size: 100, 100 AnchorLayout: anchor_x: 'right' anchor_y: 'bottom' InfoOptionsPanel: size: 100, 100  main.py  from kivy.app import App from kivy.lang.builder import Builder from kivy.uix.boxlayout import BoxLayout from kivy.uix.stacklayout import StackLayout import os kivy_design_files = ["Info_options", "Info", "main_screen",] for kv_file in kivy_design_files: Builder.load_file(os.path.join(os.path.abspath(os.getcwd()), kv_file + ".kv")) class InfoPanel(BoxLayout): pass class InfoOptionsPanel(StackLayout): def on_press(self): print("Button pressed\n"*55) class MainPanel(BoxLayout): pass class MyApp(App): def build(self): return MainPanel() if __name__ == '__main__': MyApp().run() 

Answer: You don't need to inherit from Kivy's layout classes in both main.py and kivy files. Either remove inheritance from main.py or from your kivy files. For main.py, class InfoPanel(): # don't inherit from BoxLayout pass class InfoOptionsPanel(): # don't inherit from StackLayout def on_press(self): print("Button pressed\n"*55) class MainPanel(): # don't inherit from BoxLayout pass  Or for kivy files (do the same in each), <InfoOptionsPanel>: # removed @StackLayout orientation: 'lr-tb' Button: text: 'Button' on_press: root.on_press() size_hint :.7, 0.1  I would suggest you to remove inheritance from main.py and keep python code nice and clean. Kivy files are meant to deal with UI, so take advantage of that. Also, use root.on_press() instead of self.parent.on_press(). Both will work regardless, but root clarifies that you're referring to the root widget. self.parent would refer to some other widget if you add multiple parent widgets or layouts to Button in the future.
Question: E.g. DF which contains number of executions across timestamps.  DateTime Execution 0 2023-04-03 07:00:00 4 1 2023-04-03 10:00:00 1 2 2023-04-03 12:00:00 1 3 2023-04-03 14:00:00 1 4 2023-04-03 18:00:00 1 <class 'pandas.core.frame.DataFrame'> RangeIndex: 5080 entries, 0 to 5079  Below is the output I'm trying to achieve  DateTime Execution 0 2023-04-03 07:00:00 1 1 2023-04-03 08:00:00 1 2 2023-04-03 09:00:00 1 3 2023-04-03 10:00:00 1 4 2023-04-03 10:00:00 1 5 2023-04-03 12:00:00 1 6 2023-04-03 14:00:00 1 7 2023-04-03 18:00:00 1  So want to distribute the each executions more than 1 to evenly hourly timestamp. I tried: How to divide up 'supply' evenly among rows in a dataframe by rank in Python? but this doesn't give the desired output. tried this but it is only for arranging evenly arrange dataframe rows based on the values in a given column

Answer: Use Index.repeat with DataFrame.loc for repeat rows, set 1 and add hours by to_timedelta with GroupBy.cumcount: #if string repr of datetimes df['DateTime'] = pd.to_datetime(df['DateTime']) out = (df.loc[df.index.repeat(df['Execution'])] .assign(Execution=1, DateTime = lambda x: x['DateTime'] + pd.to_timedelta(x.groupby(level=0).cumcount(), unit='H')) .reset_index(drop=True)) print (out) DateTime Execution 0 2023-04-03 07:00:00 1 1 2023-04-03 08:00:00 1 2 2023-04-03 09:00:00 1 3 2023-04-03 10:00:00 1 4 2023-04-03 10:00:00 1 5 2023-04-03 12:00:00 1 6 2023-04-03 14:00:00 1 7 2023-04-03 18:00:00 1
Question: Consider a date-indexed DataFrame: d0 = datetime.date(2024,5,5) d1 = datetime.date(2024,5,10) df0 = pd.DataFrame({"a":[1,2],"b":[10,None],"c":list("xy")}, index=[d0,d1]) df0.index Index([2024-05-05, 2024-05-10], dtype='object')  Note that df0.index.dtype is object. Now, lookup works for date: df0.loc[d0] a 1 b 10.0 c x Name: 2024-05-05, dtype: object  but both df0.loc[str(d0)] and df0.loc[pd.Timestamp(d0)] raise KeyError. This seems to be reasonable. However, consider df1 = df0.reindex(pd.date_range(d0,d1)) df1.index DatetimeIndex(['2024-05-05', '2024-05-06', '2024-05-07', '2024-05-08', '2024-05-09', '2024-05-10'], dtype='datetime64[ns]', freq='D')  Note that df1.index.dtype is datetime64. Now, lookup works for both df1.loc[pd.Timestamp(d0)] (as expected) and df1.loc[str(d0)] (why?!) but not for df1.loc[d0] (if it works for a string, why not date?!) Is this the expected behavior (a bug with tenure)? Is this intentional? PS. Reported.

Answer: Looking at the source code it's how pandas implements .loc for DateTimeIndex:  https://github.com/pandas-dev/pandas/blob/d9cdd2ee5a58015ef6f4d15c7226110c9aab8140/pandas/core/indexes/datetimes.py#L596  It's implemented for:  datetime.datetime and np.datetime64 - https://github.com/pandas-dev/pandas/blob/d9cdd2ee5a58015ef6f4d15c7226110c9aab8140/pandas/core/arrays/datetimes.py#L211C28-L211C51 str datetime.timedelta - raises TypeError datetime.time  But NOT for datetime.date
Question: I am trying to make a countdown. Now when I use a while loop to achieve that, I also need to print count down, but what happens is that the countdown takes place in separate lines printing the text over and over again. I want the loop to print the text only once but keep changing the value of countdown until countdown is over. My code is:  clearing = True countdown = 3 while clearing: while countdown > 0: print(f"Clearing console in {countdown} second(s)") time.sleep(1) countdown -= 1 os.system('clear') clearing = False  and this is the result I am currently achieving in the console rn: Clearing console in 3 second(s) Clearing console in 2 second(s) Clearing console in 1 second(s)  see the print repeats evertime until countdown == 0 what I want is for the loop to print the text only once but keep changing the value of countdown inside the text something like this: Clearing console in '3 then 2 then 1' second(s) 

Answer: I am posting the answer from the comments:  clearing = True countdown = 3 while clearing: while countdown > 0: print(f"Clearing console in {countdown} second(s)", end='\r', flush=True) time.sleep(1) countdown -= 1 os.system('clear') clearing = False  I also found another way to achieve this effect: clearing = True countdown = 3 while clearing: while countdown > 0: os.system('cls') print(f"Exiting in {countdown} second(s)") time.sleep(1) countdown -= 1 os.system('cls') os.system('cls') clearing = False
Question: I have a Dataframe in Python whith NaN, as this: import pandas as pd details = { 'info1' : [10,None,None,None,None,None,15,None,None,None,5], 'info2' : [15,None,None,None,10,None,None,None,None,None,20], } df = pd.DataFrame(details) print(df)      info1 info2     0 10 15   1 nan nan   2 nan nan   3 nan nan   4 nan 10   5 nan nan   6 15 nan   7 nan nan   8 nan nan   9 nan nan   10 5 20    How to replace NaNs with the random number (e.g., uniform) in a specific range (based on rows that have values), as this: 

Answer: For a vectorial solution, directly call np.random.uniform with the ffill/bfill as boundaries: import numpy as np df[:] = np.random.uniform(df.ffill(), df.bfill())  Output (with np.random.seed(0)):  info1 info2 0 10.000000 15.000000 1 13.013817 12.275584 2 12.118274 11.770529 3 12.187936 10.541135 4 14.818314 10.000000 5 13.958625 15.288949 6 15.000000 19.255966 7 14.289639 10.871293 8 14.797816 18.326198 9 7.218432 18.700121 10 5.000000 20.000000  Note As pointed out by @wjandrea, the behavior of uniform is not officially supported when low > high*. If you want a robust solution, use my original approach with an intermediate array and sort: import numpy as np # ensure the low boundary is before the high tmp = np.sort(np.dstack([df.ffill(), df.bfill()]), axis=2) # generate random numbers between low and high df[:] = np.random.uniform(tmp[..., 0], tmp[..., 1])  (*)  If high < low, the results are officially undefined and may eventually raise an error, i.e. do not rely on this function to behave when passed arguments satisfying that inequality condition.