eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I currently have a regex that selects all occurrences of (, , , );: (\s\()|(,\s)|(\),)|(\);) however, I've been trying to figure out a way so that if anything is between single quotes 'like this, for example', it'll ignore any of the matches listed above. I tried many different solutions, however none of them seemed to work for me. Does anyone know of ways I could make this work?

Answer: Expressing a not-find is a tricky thing, as everything around a regex is designed to work in a positive / greedy way (find as much as possible, whenever somehow possible). The easiest and most likely fastest thing you could to is to remove the parts you want to exclude prior to applying your search, assuming quotes always appear in pairs: "'[^']*'" => ""  and then apply your search to the remaining string. If the string needs to be modified "inplace", you could first search for these things and replace them with arbitrary, non-colliding placeholders that do not appear naturally, and replace them later again. (I quite often use something like ###Placeholder1### or something for that purpose. Easy to match and replace again, and almost guaranteed to not appear elsewhere naturally). Python example: import re text = "this is a , and this a ( whith a ) while 'this ( is in quotes,therefore excluded' unlike these: ( ) , but 'these () are again'. period." print(text) placeholders = [] def repl(m): contents = m.group(1) placeholders.append(contents) return "###Placeholder{0}###".format(len(placeholders) - 1) temp=re.sub('(\'[^\']*\')', repl, text) print(temp) temp=re.sub('([,\)\(])', "`\\1`", temp) print(temp) for k in range(len(placeholders)): temp = re.sub("###Placeholder{0}###".format(k), placeholders[k], temp) print(temp)  (Note that the ### also ensures that Placeholder1 and Placeholder13 won't collide later on.)  this is a , and this a ( whith a ) while 'this ( is in quotes,therefore excluded' unlike these: ( ) , but 'these () are again'. period. this is a , and this a ( whith a ) while ###Placeholder0### unlike these: ( ) , but ###Placeholder1###. period. this is a , and this a ( whith a ) while ###Placeholder0### unlike these: ( ) , but ###Placeholder1###. period. this is a , and this a ( whith a ) while 'this ( is in quotes,therefore excluded' unlike these: ( ) , but 'these () are again'. period.   Or with the pythonic * operator, the final round of re-replacing could be omitted. (This however may cause issues if {0} and stuff appear naturally): import re text = "this is a , and this a ( whith a ) while 'this ( is in quotes,therefore excluded' unlike these: ( ) , but 'these () are again'. period." print(text) placeholders = [] def repl(m): placeholders.append(m.group(1)) return "{"+"{0}".format(len(placeholders) - 1) + "}" temp=re.sub('(\'[^\']+\')', repl, text) print(temp) temp=re.sub('([,\)\(])', "`\\1`", temp) print(temp) temp = temp.format(*placeholders) print(temp)
Question: Suppose I have a function: def f(prev, curr): return prev * 2 + curr  (Just an example, could have been anything) And a Polars dataframe: | some_col | other_col | |----------|-----------| | 7 | ... | 3 | | 9 | | 2 |  I would like to use f on my dataframe cumulatively, and the output would be: | some_col | other_col | |----------|-----------| | 7 | ... | 17 | | 43 | | 88 |  I understand that, naturally, this type of calculation isn't going to be very efficient since it has to be done one row at a time (at least in the general case). I can obviously loop over rows. But is there an elegant, idiomatic way to do this in Polars?

Answer: It depends on the exact operation you need to perform. The example you've given can be expressed in terms of .cum_sum() with additional arithmetic: def plus_prev_times_2(col): x = 2 ** pl.int_range(pl.len() - 1).reverse() y = 2 ** pl.int_range(1, pl.len()) cs = (x * col.slice(1)).cum_sum() return cs / x + col.first() * y df = pl.DataFrame({"some_col": [7, 3, 9, 2]}) df.with_columns( pl.col.some_col.first() .append(pl.col.some_col.pipe(plus_prev_times_2)) .alias("plus_prev_times_2") )  shape: (4, 2) ┌──────────┬───────────────────┐ │ some_col ┆ plus_prev_times_2 │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞══════════╪═══════════════════╡ │ 7 ┆ 7.0 │ │ 3 ┆ 17.0 │ │ 9 ┆ 43.0 │ │ 2 ┆ 88.0 │ └──────────┴───────────────────┘  Vertical fold/scan In general, I believe what you're asking for is called a "Vertical fold/scan"  https://github.com/pola-rs/polars/issues/12165  Polars only offers a horizontal version, pl.cum_fold df = pl.DataFrame(dict(a=[7], b=[3], c=[9], d=[2])) df.with_columns( pl.cum_fold(acc=0, function=lambda acc, x: acc * 2 + x, exprs=pl.all()) )  shape: (1, 5) ┌─────┬─────┬─────┬─────┬──────────────┐ │ a ┆ b ┆ c ┆ d ┆ cum_fold │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 ┆ i64 ┆ struct[4] │ ╞═════╪═════╪═════╪═════╪══════════════╡ │ 7 ┆ 3 ┆ 9 ┆ 2 ┆ {7,17,43,88} │ └─────┴─────┴─────┴─────┴──────────────┘  As discussed in the issue, a vertical equivalent would be hugely inefficient. For an efficient approach, you can write plugins in Rust:  https://marcogorelli.github.io/polars-plugins-tutorial/cum_sum/  But using something like numba is probably easier to implement. There are several existing numba answers, e.g.  Python (Polars): Vectorized operation of determining current solution with the use of previous variables
Question: I have been following a machine-learning book by Chollet and I keep getting this error in this block of code, specifically in the 3rd line. It seems I am passing a Keras tensor into a tf function but I don't know how to get around this. import tensorflow as tf inputs = keras.Input(shape=(None,), dtype="int64") embedded = tf.one_hot(inputs, depth=max_tokens) x = layers.Bidirectional(layers.LSTM(32))(embedded) x = layers.Dropout(0.5)(x) outputs = layers.Dense(1, activation="sigmoid")(x) model = keras.Model(inputs, outputs) model.compile(optimizer="rmsprop", loss="binary_crossentropy", metrics=["accuracy"]) model.summary()  I tried to follow the solution in the error message which told me to create a new class but didn't know how to do it.

Answer: Wrap the TensorFlow function (tf.one_hot) in a layer. And, replace the call to the function with a call to the new layer. eg: class EmbeddedLayer(keras.Layer): def call(self, x): return tf.one_hot(x, depth=max_tokens) inputs = keras.Input(shape=(None,), dtype="int64") embedded = EmbeddedLayer()(inputs) x = layers.Bidirectional(layers.LSTM(32))(embedded) x = layers.Dropout(0.5)(x) outputs = layers.Dense(1, activation="sigmoid")(x) model = keras.Model(inputs, outputs) model.compile( optimizer="rmsprop", loss="binary_crossentropy", metrics=["accuracy"], ) model.summary()
Question: When I run a script from the Spyder console using runfile, source file lines containing expressions (e.g., "HELLO") don't print out to the console. I have to explicitly print, e.g., print("HELLO"). Is there a way to output the string representation of a pandas DataFrame using a method? Something like the very last chained method below: dfShipName[['ShpNm','ShipNamesPrShpNm']] \ .drop_duplicates() \ .groupby('ShipNamesPrShpNm',as_index=False)['ShipNamesPrShpNm'] \ .agg(['count']) \ .rename(columns={'count':'nShpNm'}) \ .sort_values(by='nShpNm',ascending=False) \ .head(20).print()  I could encapsulate the entire thing (sans .print()) in a print() invocation, but that's another level of indentation. Just appending a print()-like method avoids having to be much more cleaner, which makes a big difference when I have lots of these (for exploratory analysis). I can also toggle the output by simply commenting away the .print().

Answer: I would just save the dataframe to a variable and print the variable. In any case, you can use pipe to achieve the behavior you're looking for: import pandas as pd df = pd.DataFrame({"a": [1, 2, 3]}) df.add(4).pipe(print)   a 0 5 1 6 2 7
Question: I'm using ollama to stream a response from llama2 large language model. The functionality I need is, when I click the stop button, it should stop the thread immediately. The code below works but the problem is, it always waits for the first chunk to be available before it checks if stop is True. Sometimes the response can take some time before the first chunk is ready. It shouldn't wait for the first chunk. It should immediately stop the thread when stop button is clicked. My code: import tkinter as tk import ollama import threading stop = False # Create the main window root = tk.Tk() root.title("Tkinter Button Example") def get_answer(): print("Start") stream = ollama.chat( model='llama2', messages=[{'role': 'user', 'content': 'write a story about earth'}], stream=True, ) for chunk in stream: if stop is False: print(chunk['message']['content'], end='', flush=True) else: print("Stopped") return # Define the function to be called when the button is pressed def on_button_click(): global stop stop = True print("Button clicked!") # Create the button button = tk.Button(root, text="Stop", command=on_button_click) # Place the button on the window button.pack(pady=20) thread = threading.Thread(target=get_answer) thread.start() # Start the Tkinter event loop root.mainloop() 

Answer: You cannot instantly terminate a thread in python. This ollama API currently offers an async client, you can use the async client and cancel the Task, this should close the async connection almost instantly. import tkinter as tk import ollama import threading import asyncio from typing import Optional # Create the main window root = tk.Tk() root.title("Tkinter Button Example") client = ollama.AsyncClient() async def get_answer(): print("Start") stream = await client.chat( model='llama2', messages=[{'role': 'user', 'content': 'write a story about earth'}], stream=True, ) async for chunk in stream: print(chunk['message']['content'], end='', flush=True) worker_loop: Optional[asyncio.AbstractEventLoop] = None task_future: Optional[asyncio.Future] = None def worker_function(): global worker_loop, task_future worker_loop = asyncio.new_event_loop() task_future = worker_loop.create_task(get_answer()) worker_loop.run_until_complete(task_future) # Define the function to be called when the button is pressed def on_button_click(): # the loop and the future are not threadsafe worker_loop.call_soon_threadsafe( lambda: task_future.cancel() ) print("Button clicked!") # Create the button button = tk.Button(root, text="Stop", command=on_button_click) # Place the button on the window button.pack(pady=20) thread = threading.Thread(target=worker_function) thread.start() # Start the Tkinter event loop root.mainloop()  this does raise an asyncio.CancelledError in the worker thread, so you may want to catch it if you don't want errors on your screen, and you may want to wrap the whole thing in a class rather than relying on globals.
Question: I`ve been working on my code for Micropython-flashed ESP8266, and well... this error occured: from time import perf_counter   ImportError: can't import name perf_counter  I believe that the whole code itself is irrelevant, as the problem occurs during the import itself. What may be relevant though, is the list of functions imported, which looks strange to me: >>> import time >>> dir(time) ['__class__', '__name__', '__dict__', 'gmtime', 'localtime', 'mktime', 'sleep', 'sleep_ms', >'sleep_us', 'ticks_add', 'ticks_cpu', 'ticks_diff', 'ticks_ms', 'ticks_us', 'time', 'time_ns']  I'm currently using Python version 3.9.0 I have tried looking up the failure on google, but found only the older failures that are associated with time.clock removal and its later replacement with time.perf_counter.

Answer: In micropython, according to its documentation the time module doesn't have perf_counter.
Question: I have a large array and I want to mask out certain values (set them to nodata). But I'm experiencing an out-of-memory error despite having sufficient RAM. I have shown below an example that reproduces my situation. My array is 14.5 GB and the mask is ~7GB, but I have 64GB of RAM dedicated to this, so I don't understand why this fails. import numpy as np arr = np.zeros((1, 71829, 101321), dtype='uint16') arr.nbytes #14555572218 mask = np.random.randint(2, size=(71829, 101321), dtype='bool') mask.nbytes #7277786109 nodata = 0 #this results in OOM error arr[:, mask] = nodata  Interestingly, if I do the following, then things work. arr = np.zeros((71829, 101321), dtype='uint16') arr.nbytes #14555572218 mask = np.random.randint(2, size=(71829, 101321), dtype='bool') mask.nbytes #7277786109 nodata = 0 #this works arr[mask] = nodata  But it isn't something I can use. This code will be a part of a library module that would need to accept a variable value for the zeroth dimension. My guess is that arr[mask] = nodata is modifying the array in-place but arr[:, mask] = nodata is creating a new array, but I don't know why that would be the case. Even if it did, there should still be enough space for that, since the total size of arr and mask would be 22GB and I have 64GB of RAM. I tried searching about this, I found this but I'm new to numpy and I didn't understand the explanation of the longer answer. I did try the np.where approach from the other answer to that question, but I still get OOM error. Any input would be appreciated.

Answer: I suspect the issue here is that combining slice-based and mask-based indexing leads to a memory-inefficient codepath. You might try expressing it this way so that you're using entirely mask-based indexing: arr[mask[None]] = nodata  I don't know enough about the implementation of np.ndarray.__setitem__ to guess at why the arr[:, mask] version leads to memory issues.
Question: I'm trying to represent a special square ware characterized by some feature as represented here:  all values are related to thick parameter. Tails are related to the thick parameter and to the total length (the size of the tail is equal to three times the thickness plus a remainder which depends on the number of notches that can be placed in the total length). here is the code that allowed me to obtain some good results: import numpy as np thick = 4 length = 60 gap = 2*thick remain = length % thick tail = 3*thick + remain half_floor_division = int(((length-tail) // (thick*2))) motif = np.full(half_floor_division,gap) motif = np.concatenate([[tail],motif[:-1],[tail]]) x = np.cumsum(motif) x = np.insert(x, 0, 0., axis=0) x = np.repeat(x,2 )[1:-1] y = np.tile([0,0,thick,thick],int(np.ceil(len(x)/4)))[:len(x)]  and here are some results obtained with different lengths and thicknesses:  Well the code seems a little complicated to me... I have the impression that there is a way to make it simpler... and in addition it is unstable since for certain values ​​the end of the pattern is no longer correct. Anyone have any advice? or another approach to correct the situation?

Answer: In your "OK" examples the parts do not add up to the length... I don't fully understand what numpy array you are trying to build, but to compute the values of the parameters, I think the easiest way is to take away from length the two default tails and one gap. What's left will be an unknonwn number of gap pairs including a valley and a crest, plus the two remains to add to each tail. In code, this would look like: def get_motif(length, thick): gap = 2 * thick tail = 3 * thick n, remain = divmod(length - 2 * tail - gap, 2 * gap) tail += remain / 2 motif = [tail] + [gap] * (2 * n + 1) + [tail] assert sum(motif) == length return motif  Running this on your first and last examples: >>> get_motif(60, 4) [18.0, 8, 8, 8, 18.0] >>> get_motif(75, 3) [10.5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 10.5]
Question: I have some json data (sample below). aws glue crawler reads this data and creates a glue catalog database with table , and sets the date field as a string field . is there a way , i can format date in my json file such that crawler can identify this as a date field ? I plan to read this data into dynamic frame via aws glue etl and push it to a sql database , where I want to save it as a date field , so that it is easy to query and do comparisons on the date field. example of script below. can i convert the string date field to rds date field in spark data frame? myscript.py data=gluecontext.create_dynamic_frame.from_catalog(database="sample", table_name="table" ... data_frame=data.toDF() //convert the string field to date field in the spark data frame  {"id": "abc", .... date="2024-07-09"} ... 

Answer: You can use to_date to convert the string field to the date field in the spark dataframe as follows: from pyspark.sql.functions import to_date data=gluecontext.create_dynamic_frame.from_catalog(database="sample", table_name="table") data_frame = data.toDF() # convert the string field to the date field in the spark data frame data_frame = data_frame.withColumn("date", to_date("date", "yyyy-MM-dd"))
Question: I'd like to do the following in python using the polars library: Input: df = pl.from_repr(""" ┌──────┬────────┐ │ Name ┆ Number │ │ --- ┆ --- │ │ str ┆ i64 │ ╞══════╪════════╡ │ Mr.A ┆ 1 │ │ Mr.A ┆ 4 │ │ Mr.A ┆ 5 │ │ Mr.B ┆ 3 │ │ Mr.B ┆ 5 │ │ Mr.B ┆ 6 │ │ Mr.B ┆ 10 │ └──────┴────────┘ """)  Output: shape: (7, 3) ┌──────┬────────┬──────────┐ │ Name ┆ Number ┆ average │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ f64 │ ╞══════╪════════╪══════════╡ │ Mr.A ┆ 1 ┆ 0.0 │ │ Mr.A ┆ 4 ┆ 1.0 │ │ Mr.A ┆ 5 ┆ 2.5 │ │ Mr.B ┆ 3 ┆ 0.0 │ │ Mr.B ┆ 5 ┆ 3.0 │ │ Mr.B ┆ 6 ┆ 4.0 │ │ Mr.B ┆ 10 ┆ 4.666667 │ └──────┴────────┴──────────┘  That is to say:  For every first entry of a person, set the average to zero. For every subsequent entry, calculate the average based on the previous entries  Example: Mr. A started off with average=0 and the Number=1. Then, Mr. A has the Number=4, thus it took the average of the previous entry (1/1 data=1) Then, Mr. A has the Number=5, thus the previous average was: (1+4) / (2 data) = 5/2 = 2.5 And so on I've tried the rolling mean function (using a Polars Dataframe, df), however, I'm restricted by rolling_mean's window size (i.e. it calculates only the past 2 entries, plus it averages the current entry as well; I want to average only the previous entries) Does anyone have an idea? Much appreciated!: df.group_by("Name").agg(pl.col("Number").rolling_mean(window_size=2)) 

Answer:  cum_sum() and cum_count() to calculate cumulative average. shift() so it's calculated for 'previous' row. over() to do the whole operation within Name.  df.with_columns( (pl.col.Number.cum_sum() / pl.col.Number.cum_count()) .shift(1, fill_value=0) .over("Name") .alias("average") ) ┌──────┬────────┬──────────┐ │ Name ┆ Number ┆ average │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ f64 │ ╞══════╪════════╪══════════╡ │ Mr.A ┆ 1 ┆ 0.0 │ │ Mr.A ┆ 4 ┆ 1.0 │ │ Mr.A ┆ 5 ┆ 2.5 │ │ Mr.B ┆ 3 ┆ 0.0 │ │ Mr.B ┆ 5 ┆ 3.0 │ │ Mr.B ┆ 6 ┆ 4.0 │ │ Mr.B ┆ 10 ┆ 4.666667 │ └──────┴────────┴──────────┘
Question: I am trying to solve a problem some what very similar to: https://platform.stratascratch.com/coding/10362-top-monthly-sellers?code_type=2 here is my data frame:  product seller market total_sales books s1 de 10 books s2 jp 20 books s3 in 30 books s1 de 25 books s5 in 15 books s1 us 12 books s2 uk 10 clothing s1 de 11 clothing s2 de 18 clothing s1 uk 55 clothing s3 in 31 clothing s1 de 14 clothing s2 de 10 clothing s1 de 35 electronics s1 us 18 electronics s1 de 12 electronics s2 in 16 electronics s3 uk 24 electronics s1 us 37 electronics s4 jp 27 electronics s3 uk 26 electronics s1 us 15  Expected output is: product seller market total_sales books s1 de 35 books s3 in 30 books s2 jp 20 Clothing s1 de 60 Clothing s1 uk 55 Clothing s3 in 31 electronics s1 us 70 electronics s3 uk 50 electronics s4 jp 27  I want to get top three product category sales in the world. I was able to aggregate total sales basing on product,seller & group them. After grouping I am able to sort it by total_sales per group. But, I was not able to get only the top-3 total_sale rows per ['product'] group. Cany anyone help on this. import pandas as pd df = df.groupby(['product', 'seller','market'], as_index = False).agg({'total_sales':sum}) df = df.sort_values(['product','total_sales'],ascending=False).groupby(['product', 'seller','market',],as_index = False).aggregate(lambda x: ','.join(map(str, x))) print(df) 

Answer: You can do it like this: (dfs:= df.groupby(['product', 'seller', 'market'], as_index=False)['total_sales'].sum())\ .loc[dfs.groupby('product')['total_sales'].rank(ascending=False)<4]\ .sort_values(['product','total_sales'], ascending=[True, False])  Output:  product seller market total_sales 0 books s1 de 35 4 books s3 in 30 2 books s2 jp 20 6 clothing s1 de 60 7 clothing s1 uk 55 9 clothing s3 in 31 11 electronics s1 us 70 13 electronics s3 uk 50 14 electronics s4 jp 27  Details:  Use walrus operator to assign groupby product, seller and market dataframe to a variable dfs.  Filter that dataframe by dfs groupby product and rank, then boolean index dfs for only to top three rank  Lastly, sort the filtered dataframe on product and total_sales descending.
Question: I have different (X,Y) data series drawn as scatter plots on one figure. Within each series, the marker size is set to different values according to the rank of the data point within the series (i.e. the 1st point in the series is shown with a smaller marker than the 2nd, which itself is shown with a smaller marker than the 3rd and so on…). I want to show in the legend which marker size correspond to which rank. I managed to set the marker size in the legend handles to the same values as those on the graph. I also set the markerscale keyword of the legend() function to 1, aiming to have the markers be the same size in both legend and graph. However, the markers appear significantly larger in the legend than in the graph. Here is a code snippet replicating my issue. import numpy as np import numpy.random as rd import matplotlib.pyplot as plt from matplotlib.lines import Line2D rd.seed(12345) fig,ax = plt.subplots() # Number of time steps in each series nb_steps = 5 t = np.linspace(1,nb_steps,nb_steps) # Series 1 x1 = rd.random(nb_steps) y1 = rd.random(nb_steps) # Series 2 x2 = 1+2*rd.random(nb_steps) y2 = 1+rd.random(nb_steps) # Plotting with increasing marker size within series marker_sizes = (t+3)**2 ax.scatter(x1,y1, marker = '+', s = marker_sizes) ax.scatter(x2,y2, marker = '+', s = marker_sizes) # Writing marker size legend handles = [] for size, step in zip(marker_sizes, t): handles.append(Line2D([0], [0], marker='+', lw = 0, color='k', markersize=size, label=step)) ax.legend(handles = handles, markerscale=1) plt.show()  And here is the output figure Thanks in advance for your help.

Answer: The markersizes in pyplot.scatter use the squared form, proportional to area (as you have already noted). However, those in the legend appear not to. I suggest that you set your marker sizes as the linear form (so that they are OK for the legend): marker_sizes = t+3  and then make the squaring explicit in pyplot.scatter() (but NOT in the legend): ax.scatter(x1,y1, marker = '+', s = marker_sizes**2)