eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am making a modulatory function that will take keywords with special characters (@&\*%) and keep them intact while all other punctuation is deleted from a sentence. I have devised a solution, but it is very bulky and probably more complicated than it needs to be. Is there a way to do this, but in a much simpler way? In short, my code matches all instances of the special words to find the span. I then match the characters to find their span, and then I loop over the list of matches and remove any characters that also exist in the span of the found words. Code: import re from string import punctuation sentence = "I am going to run over to Q&A and ask them a ton of questions about this & that & that & this while surfacing the internet! with my raccoon buddy @ the bar." # my attempt to remove punctuation class SentenceHolder: sentence = None protected_words = ["Q&A"] def __init__(sentence): self.sentence = sentence def remove_punctuation(self): for punct in punctuation: symbol_matches: List[re.Match] = [i for i in re.finditer(punct, self.sentence)] remove_able_matches = self._protected_word_overlap(symbol_matches) for word in reversed(remove_able_word_matches): self.sentence = (self.modified_string[:word.start()] + " " + self.sentence[word.end():]) def _protected_word_overlap(symbol_matches) protected_word_locations = [] for protected_word in self.protected_words : protected_word_locations.extend([i for i in re.finditer(protected_word, self.sentence)]) protected_matches = [] for protected_word in protected_word_locations: for symbol_inst in symbol_matches: symbol_range: range = range(symbol_inst.start(), symbol_inst.end()) protested_word_set = set(range(protected_word.start(), protected_word.end())) if len(protested_word_set.intersection(symbol_range)) != 0: protected_matches.append(symbol_inst) remove_able_matches = [sm for sm in symbol_matches if sm not in protected_matches] return remove_able_matches  The output of the code: my_string = SentenceHolder(sentence) my_string.remove_punctuation()  Result: "I am going to run over to Q&A and ask them a ton of questions about this that that this while surfacing the internet with my raccoon buddy the bar"  I tried to use regex and pattern to identify all the locations of the punctuation, but the pattern I use in re.sub does not work similarly in re.match.

Answer: probably not the best, but really simple protected = ["Q&A", "stack@exchange"] protected_dict = {f'protected{i}': p_word for i, p_word in enumerate(protected)} sentence = "I am going to run over to Q&A stack@exchange and ask them a ton of questions about this & that & that & this while surfacing the internet! with my raccoon buddy @ the bar." # protect for k, v in protected_dict.items(): sentence = sentence.replace(v, k) # replace stuff sentence = sentence.replace('&', '') sentence = sentence.replace('@', '') # revert back protected words for k, v in protected_dict.items(): sentence = sentence.replace(k, v) print(sentence) # I am going to run over to Q&A stack@exchange and ask them a ton of questions about this that that this while surfacing the internet! with my raccoon buddy the bar.
Question: Consider the following example series. s = pl.Series('s', [[1, 2, 3], [3, 4, 5]])  I'd like to replace all 3s with 10s to obtain the following. res = pl.Series('s', [[1, 2, 10], [10, 4, 5]])  Is it possible to efficiently replace elements in the lists of a List column in polars? Note. I've already tried converting to a dataframe and using pl.when().then(), but pl.when() fails for input of type List[bool]. Moreover, I've experimented with pl.Expr.list.eval, but couldn't get much further than the original mask.

Answer: In the case of a simple replacement, you can also consider using pl.Expr.replace within a pl.Series.list.eval context. This approach is a bit less general, but more concise than a pl.when().then().otherwise() construct. s.list.eval(pl.element().replace({3: 10}))  shape: (2,) Series: 's' [list[i64]] [ [1, 2, 10] [10, 4, 5] ]
Question: One of the new features in python 3.13 is a more sophisticated interactive interpreter. However, it seems that this interpreter is no longer based on readline, and therefore no longer respects ~/.inputrc. I particularly miss the set editing-mode vi behavior. Is there a way to get this same "vi mode" behavior with the python 3.13 interpreter (other than just disabling the new interpreter completely with PYTHON_BASIC_REPL=1)? Caveat: The specific version of python I'm using is 3.13.0b3. This is a beta release, so I'd be satisfied if the answer is "just wait for the actual release of python 3.13; this feature is planned but hasn't been implemented yet".

Answer: According to this CPython issue and this comment on another issue, vi editing mode is not coming any time soon to the new REPL. It is still possible to use the old REPL by setting the PYTHON_BASIC_REPL=1 environmental variable.
Question: Pylance claims that code under if not dfs: is unreachable and greys it out in VS Code. import pandas as pd def concat_dfs(dfs: tuple[pd.DataFrame | None], logger) -> pd.DataFrame | None: # filter out dfs that are None, if any dfs = [df for df in dfs if df is not None] # concatenate dfs, if any are not None: if not dfs: logger.warning("All elements of dfs are None, so no inputs remain to be analyzed") return None # greyed out as unreachable else: return pd.concat(dfs).sort_index()  I tried returning dfs alone, and not pd.concat(dfs).sort_index(), as there are/were some pd.concat related bugs in Pylance. I also tried making a little test. If Pylance is right, I would have expected not dfs = False, and it is True instead: dfs = (None, None) dfs = [df for df in dfs if df is not None] # dfs is an empty list not dfs # returns True --> code should be reachable when every dataframe in dfs is None to begin with... 

Answer: The reason the code is considered unreachable is that the type hint for the argument dfs is tuple[pd.DataFrame | None]. Type hints take precedence over code interpretation, and this type hint means that the tuple has one element, so not dfs can never be true. Perhaps the type you really want to express is tuple[pd.DataFrame | None, ...] (but you should avoid assigning an object of a different type than the annotation to a variable. In other words, you should not assign [df for df in dfs if df is not None], which is a list, to dfs.).
Question: I have an 1d numpy array of values: v = np.array([0, 1, 4, 0, 5])  Furthermore, I have a 2d numpy array of boolean masks (in production, there are millions of masks): m = np.array([ [True, True, False, False, False], [True, False, True, False, True], [True, True, True, True, True], ])  I want to apply each row from the mask to the array v, and then compute the mean of the masked values. Expected behavior: results = [] for mask in m: results.append(np.mean(v[mask])) print(results) # [0.5, 3.0, 2.0]  Easy to do sequentially, but I am sure there is a beautiful version in parallel? One solution, that I've found: mask = np.ones(m.shape) mask[~m] = np.nan np.nanmean(v * mask, axis=1) # [0.5, 3.0, 2.0]  Is there another solution, perhaps using np.ma module? I am looking for a solution that is faster than my current two solutions.

Answer: Faster Numpy code A faster Numpy way to do that is to perform a matrix multiplication: (m @ v) / m.sum(axis=1)  We can optimize this further by avoiding implicit conversion and perform the summation with 8-bit integer (this is safe only because v.shape[1] is small -- ie. less than 127): (m @ v) / m.view(np.int8).sum(dtype=np.int8, axis=1)   Even faster code with Numba multithreading We can use Numba to write a similar function and even use multiple threads: import numba as nb @nb.njit('(float32[::1], bool_[:,::1])', parallel=True) def compute(v, m): si, sj = m.shape res = np.empty(si, dtype=np.float32) for i in nb.prange(si): s = np.float32(0) count = 0 for j in range(sj): if m[i, j]: s += v[j] count += 1 if count > 0: res[i] = s / count else: res[i] = np.nan return res   Results Here are results on my i5-9600KF CPU (with 6 cores): Initial vectorized code: 136 ms jakevdp's solution: 74 ms Numpy matmul: 28 ms Numba sequential code: 21 ms Optimized Numpy matmul: 20 ms <----- Numba parallel code: 4 ms <-----  The sequential Numba implementation is unfortunately not much better than the Numpy one (Numba do not generate a very good code in this case), but it scale well so the parallel version is significantly faster. The Numba parallel version is 34 times faster than the initial vectorized code and 18 times faster than the jakevdp's solution. The Numpy optimized matrix-multiplication-based solution is 3.7 times faster jakevdp's solution. Note that the Numba code is also a bit better than the jakevdp's solution and the optimized Numpy one since it support the case where the mean is invalid (NaN) without printing a warning about a division by 0.
Question: I am currently creating a scatter plot with the results of some evaluation I am doing. To get a dataframe of the same structure as mine you can run: import pandas as pd models = ["60000_25_6", "60000_26_6"] results = [] for i in range(10): for model in models: results.append({"simulation": i, "model_id": model, "count_at_1": 1, "count_at_5": 5, "count_at_10": 10}) df = pd.DataFrame(results)  You will end up with a pandas dataframe that looks like so, just with default values (this is a smaller dataframe, note that the size is variable and much larger depending on the settings I use):  simulation model_id count_at_1 count_at_5 count_at_10 0 0 60000_25_6 60 77 84 1 0 60000_26_6 60 76 83 2 1 60000_25_6 69 80 82 ... 18 9 60000_25_6 1 70 79 19 9 60000_26_6 1 68 74  I then use the following code to add colors to each point: import matplotlib.pyplot as plt colors = plt.get_cmap('hsv') colors = [colors(i) for i in np.linspace(0,0.95, len(models))] cmap = {model: colors[i] for i, model in enumerate(models)} df['color'] = df.apply(lambda row: cmap[row['model_id']], axis=1)  And df is now:  simulation model_id count_at_1 count_at_5 count_at_10 color 0 0 60000_25_6 74 81 83 (1.0, 0.0, 0.0, 1.0) 1 0 60000_26_6 75 80 83 (1.0, 0.0, 0.5, 1.0) 2 1 60000_25_6 71 84 89 (1.0, 0.0, 0.0, 1.0) ... 18 9 60000_25_6 2 69 79 (1.0, 0.0, 0.0, 1.0) 19 9 60000_26_6 2 72 78 (1.0, 0.0, 0.5, 1.0)  However when I run: df.plot.scatter('count_at_1', 'count_at_5', c='color', legend=True) plt.show()  No legend appears I just get a normal plot like so:  How can I add a legend where it looks something like: [model_id] [color] ...  But in the normal matplotlib format, I'll take it anywhere on the plot.

Answer: The data need assigned a label for the legend, so one option is: fig, ax = plt.subplots() for model in df['model_id'].unique(): df[df["model_id"].eq(model)].plot.scatter('count_at_1', 'count_at_5', c='color', label=model, ax=ax) ax.legend(markerfirst=False)  With your sample data:
Question: I use list comprehension to load only the images from a folder that meet a certain condition. In the same operation, I would also like to keep track of those that do not meet the condition. This is where I am having trouble. If the if and else conditions are at the beginning, each iteration yields a resulting element. The items that are caught by else are replaced by None rather than being excluded from the result list. I can't figure out how to add an else condition such that an operation can be done on the items caught by else without including them in the resulting list. here is a simplified and generalized version of the code: exclude_imgs = [1, 3] final = [ n for ix, n in enumerate(sorted(["img1", "img4", "img3", "img5", "img2"])) if ix + 1 not in exclude_imgs ]  [ins] In [4]: final Out[4]: ['img2', 'img4', 'img5']  adding else condition to store excluded images: exclude_imgs = [1, 3] excluded = [] final = [ n if ix + 1 not in exclude_imgs else excluded.append(n) for ix, n in enumerate(sorted(["img1", "img4", "img3", "img5", "img2"])) ]  [ins] In [6]: final Out[6]: [None, 'img2', None, 'img4', 'img5'] [ins] In [7]: excluded Out[7]: ['img1', 'img3']  How can I write this so that the results are as follows: final: ['img2', 'img4', 'img5'] excluded: ['img1', 'img3']  ?

Answer: Consider not using a list comprehension at all, precisely because you want to create two lists, not just one. exclude_imgs = [1, 3] excluded = [] final = [] for ix, n in enumerate(sorted(["img1", "img4", "img3", "img5", "img2"]), start=1): (excluded if ix in excluded_imgs else final).append(n)
Question: I have a function which can take an np.ndarray of shape (3,) or (3, N), or (3, N, M), etc.. I want to add to the input array an array of shape (3,). At the moment, I have to manually check the shape of the incoming array and if neccessary, expand the array that is added to it, so that I don't get a broadcasting error. Is there a function in numpy that can expand my array to allow broadcasting for an input array of arbitrary depth? def myfunction(input_array): array_to_add = np.array([1, 2, 3]) if len(input_array.shape) == 1: return input_array + array_to_add elif len(input_array.shape) == 2: return input_array + array_to_add[:, None] elif len(input_array.shape) == 3: return input_array + array_to_add[:, None, None] ... 

Answer: One option would be to transpose before and after the addition: (input_array.T + array_to_add).T  You could also use expand_dims to add the extra dimensions: (np.expand_dims(array_to_add, tuple(range(1, input_array.ndim))) + input_array )  Alternatively with broadcast_to on the reversed shape of input_array + transpose: (np.broadcast_to(array_to_add, input_array.shape[::-1]).T + input_array )  Or with a custom reshape: (array_to_add.reshape(array_to_add.shape+(1,)*(input_array.ndim-1)) + input_array )
Question: I want to append different dictionaries having the key as the meal_name and having total_calories and total_protein as a tuple being the value of the dictionary without overwriting the original dictionaries under the main dictionary having date as the key value. I tried using the code below to add the new dictionary but it overwrites the previous one. date = input("Enter the date in (Day-Month-Year) format: ") name = input("Enter the name of the meal: ") current_meal[name] = (total_calories, total_protein) meal_data[date] = current_meal  This is what the dictionary looks like: {'23-07-2024': {'Smoothie': (245.0, 17.0)}}  Suppose I want to add a new meal to the existing date without overwriting the Smoothie dictionary. Is this possible.

Answer: You can use the if-else statement to check whether the current date is in the dictionary. date = input("Enter the date in (Day-Month-Year) format: ") name = input("Enter the name of the meal: ") if date in meal_data: # If date is already in meal_date then set the current_meal to existing meal current_meal = meal_data[date] else: # if not then set it to empty dict current_meal = {} current_meal[name] = (total_calories, total_protein) # Adding new meal to the current_meal meal_data[date] = current_meal  OUTPUT
Question: Here is my code: import osmnx as ox # Use the following commands to download the graph file of NYC G = ox.graph_from_place('New York City', network_type='drive', simplify=True) # Coordinates for origin and destination orig_x = 40.6662 orig_y = -73.9340 dest_x = 40.6576 dest_y = -73.9208 # Find the nearest nodes orig_node = ox.distance.nearest_nodes(G, orig_x, orig_y, return_dist=True) dest_node = ox.distance.nearest_nodes(G, dest_x, dest_y, return_dist=True) print(f"Origin node: {orig_node}, Destination node: {dest_node}") # Calculate the shortest path route = ox.shortest_path(G, orig_node, dest_node, weight='length') travel_secs = sum(G[u][v][0]['length'] for u, v in zip(route[:-1], route[1:])) * 186.411821 print(f"Travel time (seconds): {travel_secs}")  I'm trying to find the shortest path between these two points but I get networkx.exception.NodeNotFound: Either source 15056632.490169104 or target 15056625.267485507 is not in G  Sorry if this is something obvious, I'm new to this library and haven't found any good documentation on this issue.

Answer: That's because of two problems : You swapped the x/y (i.e, New York is at -73.935242/40.730610) and considered the osmid/dist as pair of coordinates when asking for the shortest_path : import osmnx as ox G = ox.graph_from_place("New York City", network_type="drive", simplify=True) orig_x, orig_y = -73.9340, 40.6662 dest_x, dest_y = -73.9208, 40.6576 orig_node = ox.distance.nearest_nodes(G, orig_x, orig_y) dest_node = ox.distance.nearest_nodes(G, dest_x, dest_y) route = ox.shortest_path(G, orig_node, dest_node, "length") travel_secs = nx.path_weight(G, route, "length") * 186.411821 # or if route is not needed # nx.shortest_path_length(G, orig_node, dest_node, "length") * 186.411821 print(f"Travel time (seconds): {travel_secs}") # 314517.75262762
Question: I want to match all cases where a hyphenated string (which could be made up of one or multiple hyphenated segments) ends in a consonant that is not the letter m. In other words, it needs to match strings such as: 'crack-l', 'crac-ken', 'cr-ca-cr-cr' etc. but not 'crack' (not hyphenated), 'br-oom' (ends in m), br -oo (last segment ends in vowel) or cr-ca-cr-ca (last segment ends in vowel). It is mostly successful except for cases where there is more than one hyphen, then it will return part of the string such as 'cr-ca-cr' instead of the whole string which should be 'cr-ca-cr-ca'. Here is the code I have tried with example data: import re dummy_data = """ broom br-oom br-oo crack crack-l crac-ken crack-ed cr-ca-cr-ca cr-ca-cr-cr cr-ca-cr-cr-cr """ pattern = r'\b(?:\w+-)+\w*[bcdfghjklnpqrstvwxyz](?<!m)\b' final_consonant_hyphenated = [ m.group(0) for m in re.finditer(pattern, dummy_data, flags=re.IGNORECASE) ] print(final_consonant_hyphenated)`  expected output: ['crack-l', 'crac-ken', 'crack-ed', 'cr-ca-cr-cr', 'cr-ca-cr-cr-cr']  current output:  ['crack-l', 'crac-ken', 'crack-ed', **'cr-ca-cr'**, 'cr-ca-cr-cr', 'cr-ca-cr-cr-cr']  (bold string is an incorrect match as it's part of the cr-ca-cr-ca string where the final segment ends in a vowel not a consonant).

Answer: You could add a negative lookahead to prevent having a hyphen after and also an idea to shorten [bcdfghjklnpqrstvwxyz](?<!m) to [a-z](?<![aeioum]). Update: Further as @Thefourthbird mentioned in the comments, as well putting the lookbehind after the word-boundary \b will result in better performance (fewer steps). \b(?:\w+-)+\w*[a-z]\b(?<![aeioum])(?!-)  See this demo at regex101 or even \b(?:\w+-)+\w+\b(?<![aeioum\d_])(?!-) (without the [a-z], using \w+ instead of \w* and also disallowing digits and underscore from \w in the lookbehind). With a possessive quantifier (using PyPI) further reduced: \b(?:\w+-)+\w++(?<![aeioum\d_])(?!-)
Question: I'm using alembic in my code to apply database migrations at application start. I'm also using Python's builtin logging lib to log to the terminal. After applying the migrations (or running any alembic command that prints to stdout it seems) my loggers stop working, though. Code: import logging import alembic.command from alembic.config import Config as AlembicConfig logging.basicConfig(level=logging.DEBUG) logger = logging.getLogger("app") logger.debug("Applying alembic migrations.") alembic_config = AlembicConfig("alembic.ini") alembic_config.attributes["sqlalchemy.url"] = connection_string alembic.command.upgrade(alembic_config, "head", tag="from_app") logger.debug("Terminating app.")  Expected output: DEBUG:app:Applying alembic migrations. INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL. DEBUG:app:Terminating app.  Actual output: DEBUG:app:Applying alembic migrations. INFO [alembic.runtime.migration] Context impl PostgresqlImpl. INFO [alembic.runtime.migration] Will assume transactional DDL.  The last line is missing. I've tried setting the log level again after applying the migrations (I thought maybe it changed the root logger log level): ... alembic.command.upgrade(alembic_config, "head", tag="from_app") logger.setLevel(logging.DEBUG) logger.debug("Terminating app.")  In fact, even logger.critical("blah") won't log anything anymore. I've also tried applying the basic config again and getting the logger again: ... alembic.command.upgrade(alembic_config, "head", tag="from_app") logging.basicConfig(level=logging.DEBUG) logger = logging.getLogger("app") logger.debug("Terminating app.")  But to no avail. Even the root logger isn't logging anymore: ... alembic.command.upgrade(alembic_config, "head", tag="from_app") logging.basicConfig(level=logging.DEBUG) logging.debug("Terminating app.")  Is there anything I can do to make sure that my loggers are logging? I'd like to keep using the builtin logging functionality, but I'm also open to using some lib for that.

Answer: The answer will be found in your alembic.ini file. You're running a command which was intended to be a script, not an API call, so alembic is configuring its own logging using logging.fileConfig, which by default uses disable_existing_loggers=True. That option will disable any existing non-root loggers unless they or their ancestors are explicitly named in the logging configuration file. So, the path of least resistance will be to setup your logging configuration in there too. There will be a section with the logging configuration in alembic.ini - look for a [loggers] section header. You'll want to modify the content so that your own loggers remain visible - add a [logger_app] section with the desired handlers, formatters etc. For consistency's sake, you may want to switch to using fileConfig from your own script too, instead of the logging.basicConfig. An alternative option would be to run alembic scripts in a subprocess, so that their logging configuration doesn't concern you. The stdout/stderr of the subprocess can always be captured and re-emitted as log events from your main process.
Question: I tried this: app_ui = ui.page_fluid( ui.output_text("the_txt") ) def server(input, output, session): @render.text def the_txt(): url = 'https://stackoverflow.com' clickable_url = f'<a> href="{url}" target="_blank">Click here</a>' return ui.HTML(clickable_url)  But the displayed text is the raw HTML: <a> href="https://stackoverflow.com" target="_blank">Click here</a> How do I display a clickable link in a Shiny for Python app?

Answer: In your app you have a small syntax error within the tag, you need <a ...> instead of <a> .... Below are two variants, depending on a static case or a situation where you have to render something. Static case You don't need a render function here since this is only HTML. Below are two alternatives: Either use ui.tags for creating an a tag or use ui.HTML for passing the HTML directly: from shiny import ui, App url = 'https://stackoverflow.com' app_ui = ui.page_fluid( ui.tags.a("Click here", href=url, target='_blank'), ui.p(), ui.HTML(f'<a href="{url}" target="_blank">Click here</a>') ) def server(input, output, session): return app=App(app_ui, server)   Dynamic case Here is an example where we have an input and render a link dynamically into a text. The output here is set by ui.output_ui and contains a div with the text and the url. from shiny import ui, App, render, reactive app_ui = ui.page_fluid( ui.input_select("pageSelect", label="Pages", choices=['StackOverflow', 'Google']), ui.p(), ui.output_ui("text") ) def server(input, output, session): @reactive.calc def url(): if (input.pageSelect() == "StackOverflow"): url = 'https://stackoverflow.com' else: url = 'https://google.com' return ui.tags.a("click here", href=url, target='_blank') @render.ui def text(): return ui.div("You can ", url(), " for going to ", input.pageSelect(), ".") app=App(app_ui, server)