eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I need for https://github.com/mchelem/terminator-editor-plugin to capture different type of path with line number. So far I use this pattern: (?![ab]\/)(([^ \t\n\r\f\v:\"])+?\.(html|py|css|js|txt|xml|json|vue))(\". line |:|\n| )(([0-9]+)*) I'm trying to make it work for git patch format, which adds a 'a/' and 'b/' before paths, how do I make this work, I can't make the lookahead gulp the first slash. Here's the test text: diff --git a/src/give/forms.py b/src/give/forms.py M give/locale/fr/LC_MESSAGES/django.po M agive/models/translation.py M give/views.py Some problem at src/give/widgets.py:103 Traceback (most recent call last): File "/usr/lib/python3.10/unittest/case.py", line 59, in testPartExecutor yield File "/usr/lib/python3.10/unittest/case.py", line 587, in run self._callSetUp() File "/usr/lib/python3.10/unittest/case.py", line 546, in _callSetUp self.setUp() File "/home/projects/src/give/tests/test_models.py", line 14, in setUp  https://regex101.com/r/tF50pn/1 (In this link I want the same capture text except for the first line where it is currrently capturing /src/give/forms.py and /src/give/forms.py but I want src/give/forms.py and src/give/forms.py)

Answer: You have redundant groups that you can remove and make the starting [ab]/ part optional to leave it out of first capture group. Here is refactored and optimized regex: (?:[ab]/)?([^ \t\n\r\f\v:"]+\.(?:html|py|css|js|txt|xml|json|vue))(?:". line |[:\n ])  Updated RegEx Demo
Question: I fairly know that trees are sensitive to one hot encoded (OHE) targets however I want to understand why it returns the predictions like this: array([[0, 0, 0, 0], [0, 0, 0, 0], . . . [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [0, 0, 0, 0]])  For most of the samples, it predict neither class. I will encode my targets as ordinal (since it is applicable) but what if it was not? What to do then? This is how it looks before OHE: array(['4 -8 weeks', '13 - 16 weeks', '17 - 20 weeks', ..., '9 - 12 weeks', '13 - 16 weeks'], dtype=object)  Full code: from sklearn.preprocessing import LabelBinarizer mlb = LabelBinarizer() b = mlb.fit_transform(Class) list(mlb.classes_) # Split the data into training and test sets X_train, X_test, y_train, y_test = train_test_split(data, b, test_size=0.2, random_state=42) # Create a multi-label classifier classifier = RandomForestClassifier() # Train the classifier classifier.fit(X_train, y_train) # Make predictions on the test set y_pred = classifier.predict(X_test) accuracy = accuracy_score(y_test, y_pred) 

Answer: When the targets are one-hot encoded, sklearn treats the problem as a multi-label one (each row could have any number of labels). As such, you get a predicted probability for each label, and those are independently thresholded at 0.5 in order to make the class predictions. When the targets are ordinally encoded, sklearn treats the problem as a multiclass one (each row has exactly one class). Despite the numerical ordering, sklearn doesn't care (well, except in tiebreaking) and treats the classes as unordered. The predicted probabilities sum to 1, and the predicted class is the one with largest probability. You don't need to encode labels at all. sklearn will encode them internally for computational efficiency; but leaving strings as the labels is fine, will be treated as multiclass, and allows for the class predictions to also be strings (no need to decode).
Question: In a pipe i want to use the result of previous steps as the second argument to a subsequent step. In R i can use the result of the previous chain using . like so: df %>% b(arg1b) %>% c(arg1c, .) How to do this in python for example using pipe? df.pipe(b, arg1b).pipe(c, arg1c, **)  syntax error

Answer: I'm not an active user of pandas, but this documentation page on pandas.DataFrame.pipe seems to cover your case df.pipe(b, arg1b) .pipe((c, 'second_arg_name'), arg1c)  where 'second_arg_name' should be replaced with actual name of the second argument in function c
Question: This is kind of a complicated question to explain and easier to just show as an example. So let's say I have this string here: exampleString = "abcauehafj['hello']jfa['hello']jasfjgafadsf"  And so we have the regex pattern of: regex = r"\['hello'\]"  Now, what I want to do is get the start positions of regex matches within the exampleString. This would be [10, 22], which I currently calculate using the following code: import re matches = re.finditer(regex, exampleString) start_pos = [] for match in matches: start_pos.append(match.start())  Now, the issue is, I do not want to include the length of the ['hello'] to the string itself. So the start_pos that I actually want would be [10, 13]. What would be the best way to do this without affecting the strings?

Answer: If you prefer a list comprehension, it's easy to do with enumerate, unless I misunderstood what you desired: import re exampleString = "abcauehafj['hello']jfa['hello']jasfjgafadsf" regex = r"\['hello'\]" start_pos = [match.start() - i * len(match.group()) for (i, match) in enumerate(re.finditer(regex, exampleString))]  Edit 1: of course, right after posting, I saw Kelly Bundy's comment stating exactly that... Edit 2: just for the sake of it, I tried to wrap Kelly Bundy's variable length solution into a list comprehension; it's not pretty but at least it works ;) exampleString = "abcauehafj['hi']jfa['there']jasfjgafads['hello']f" regex = r"\['[a-z]*'\]" start_pos = [match.start() - length_sum + len(match.group()) for i, match in enumerate(re.finditer(regex, exampleString)) if (length_sum := len(match.group()) + (length_sum if i else 0))]
Question: In python, how can I type hint an input that is not str or bytes and is a sequence? I have a function that I want to accept a sequence but not accept a string or bytes input. But this code allows in str and bytes: import typing def validate(input_data: typing.Sequence) -> None: # implementation that validate that input data meets expected constraints #one can ignore the implementation pass 

Answer: One can do this by creating a Sequence protocol that implements the sequence interface, and by making the contains method ingest object, str and bytes classes will not implement the new protocol. Answer from: https://github.com/python/typing/issues/256#issuecomment-1687374072  str __contains__ is incompatible with this definition bytes __contains__ is incompatible with this definition  Here's the code: import typing _T_co = typing.TypeVar("_T_co", covariant=True) class Sequence(typing.Protocol[_T_co]): """ if a Protocol would define the interface of Sequence, this protocol would NOT allow str/bytes as their __contains__ is incompatible with the definition in Sequence. methods from: https://docs.python.org/3/library/collections.abc.html#collections.abc.Collection """ def __contains__(self, value: object, /) -> bool: raise NotImplementedError def __getitem__(self, index, /): raise NotImplementedError def __len__(self) -> int: raise NotImplementedError def __iter__(self) -> Iterator[_T_co]: raise NotImplementedError def __reversed__(self, /) -> typing.Iterator[_T_co]: raise NotImplementedError def validate(input_data: Sequence) -> None: # implementation that validates that input data meets expected constraints #one can ignore the implementation pass def validate_typed_sequence(input_data: Sequence[int]) -> None: # implementation that validates that input data meets expected constraints #one can ignore the implementation pass
Question: I have the following 4 matrices: >>> a array([[0., 0.], [0., 0.]]) >>> b array([[1., 1.], [1., 1.]]) >>> c array([[2., 2.], [2., 2.]]) >>> d array([[3., 3.], [3., 3.]])  I'm creating another matrix that will contain them: >>> e = np.array([[a,b], [c,d]]) >>> e.shape (2, 2, 2, 2)  I want to "cancel the hierarchy" and reshape e into a 4x4 matrix that will look like this: 0 0 1 1 0 0 1 1 2 2 3 3 2 2 3 3  However, when I run e.reshape((4,4)), I get the following matrix: >>> e.reshape((4,4)) array([[0., 0., 0., 0.], [1., 1., 1., 1.], [2., 2., 2., 2.], [3., 3., 3., 3.]])  Is there a way to reshape my (2,2,2,2) matrix into a (4,4) matrix by cancelling the hierarchy, rather than the the by the indexing I'm currently getting?

Answer: Try np.block([[a,b],[c,d]])  This concatenates the inner lists horizontally, and does a vertical stack. Alternatively you could swap 2 axes of e and then reshape. In [41]: np.block([[a,b],[c,d]]) Out[41]: array([[0., 0., 1., 1.], [0., 0., 1., 1.], [2., 2., 3., 3.], [2., 2., 3., 3.]]) In [45]: e.transpose(0,2,1,3).reshape(4,4) Out[45]: array([[0., 0., 1., 1.], [0., 0., 1., 1.], [2., 2., 3., 3.], [2., 2., 3., 3.]])  block is doing the equivalent of: In [47]: np.vstack([np.hstack([a,b]),np.hstack([c,d])]) Out[47]: array([[0., 0., 1., 1.], [0., 0., 1., 1.], [2., 2., 3., 3.], [2., 2., 3., 3.]])
Question: Given a 2-dimensional array a, I want to update select indices specified by b to a fixed value of 1. test data: import numpy as np a = np.array( [[0, 1, 0, 0], [0, 0, 0, 0], [0, 0, 1, 0], [1, 0, 0, 0], [0, 1, 0, 1], [0, 0, 0, 0]] ) b = np.array([1, 2, 2, 0, 3, 3])  One solution is to transform b into a masked array like this: array([[0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 1, 0], [1, 0, 0, 0], [0, 0, 0, 1], [0, 0, 0, 1]])  which would allow me to do a[b.astype(bool)] = 1 and solve the problem. How can I transform b into the "mask" version below?

Answer: No need to build the mask, use indexing directly: a[np.arange(len(b)), b] = 1  Output: array([[0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 1, 0], [1, 0, 0, 0], [0, 1, 0, 1], [0, 0, 0, 1]])  That said, the mask could be built using: mask = b[:,None] == np.arange(a.shape[1])  Output: array([[False, True, False, False], [False, False, True, False], [False, False, True, False], [ True, False, False, False], [False, False, False, True], [False, False, False, True]])
Question: How can I save a table in txt file with double separator? I can save a txt file with one tab as below: df.to_csv("file1.txt", sep='\t', index=False)  Name 05:58 06:11 06:18 1234DCE -3.43 -3.427 -4.622 18DDD12 11.564 10.883 10.915 453FGH2A 0.351 0.176 0.0 8FF3EFA -0.72 0.888 -0.638 98ACD12 12.574 11.883 13.356 A2DA21 0.0 0.162 -0.162 ABC1234 -2.385 -2.348 -2.248  But need two tabs as sep. Like below: Name 05:58 06:11 06:18 1234DCE -3.43 -3.427 -4.622 18DDD12 11.564 10.883 10.915 453FGH2A 0.351 0.176 0.0 8FF3EFA -0.72 0.888 -0.638 98ACD12 12.574 11.883 13.356 A2DA21 0.0 0.162 -0.162 ABC1234 -2.385 -2.348 -2.248 

Answer: If you don't pass a filename to to_csv, Pandas will return a string. So you can add a second tab: with open('file1.txt', 'w') as fp: content = df.to_csv(sep='\t', index=False) fp.write(content.replace('\t', '\t\t').strip())  However your data will not be well aligned (row 453FGH2A for example). Maybe you should use to_string instead: df.to_string('file1.txt', col_space=16, justify='left', formatters={'Name': '{:<16}'.format}, index=False)  With to_csv: Name 05:58 06:11 06:18 1234DCE -3.43 -3.427 -4.622 18DDD12 11.564 10.883 10.915 453FGH2A 0.351 0.176 0.0 8FF3EFA -0.72 0.888 -0.638 98ACD12 12.574 11.883 13.356 A2DA21 0.0 0.162 -0.162 ABC1234 -2.385 -2.348 -2.248  With to_string: Name 05:58 06:11 06:18 1234DCE -3.430 -3.427 -4.622 18DDD12 11.564 10.883 10.915 453FGH2A 0.351 0.176 0.000 8FF3EFA -0.720 0.888 -0.638 98ACD12 12.574 11.883 13.356 A2DA21 0.000 0.162 -0.162 ABC1234 -2.385 -2.348 -2.248
Question: I have a Polars Dataframe with a mix of Series, which I want to write to a CSV / Upload to a Database. The problem is if any of the UTF8 series have non-ASCII characters, it is failing due to the DB Type I'm using so I would like to filter out the non-ASCII characters, whilst leaving everything else. I created a function that uses a lambda function, which does work, but it is slow compared with standard Polars functions and I was hoping to replace this with a Polars alternative def df_column_clean(df:pl.DataFrame, drop_non_ascii:bool=False): """ Takes a Polars Dataframe and performs data cleaning on all columns Currently it only converts string series to ascii but can be expanded in the future """ if drop_non_ascii: df_changes = [] df_columns = df.schema for col_name, col_type in df_columns.items(): if col_type != pl.Utf8: continue # Remove non-ascii characters df_changes.append(pl.col(col_name).apply(lambda x: None if x is None else x.encode('ascii', 'ignore').decode('ascii'), skip_nulls=False)) if len(df_changes) > 0: return df.with_columns(df_changes) return df  Is the method I came up with the best option or does Polars have an-inbuilt function that can be used to filter out non-ASCII characters? Thanks in advance

Answer: .str.replace_all() with a regex to match non-ascii chars: pl.col(pl.String).str.replace_all(r"[^\p{Ascii}]", "")
Question: import discord from discord.ext import commands, tasks import imageio import os import numpy as np import asyncio @stocks.command(name="graph", description="Generate and display a stock price history graph") async def graph(self, interaction: discord.Interaction, symbol): video_path = "stock_graph.mp4" imageio.mimsave(video_path, ani_frames, fps=8) message = await interaction.response.defer() asyncio.sleep(10) await interaction.followup.send(file=discord.File(video_path)) plt.close()  Error: discord.errors.NotFound: 404 Not Found (error code: 10062): Unknown interaction

Answer: First of all, understand that a Discord interaction has a short lifetime, which is exactly 3 seconds. That is, you must respond to the interaction that called your command within 3 seconds or else it will become unknown. If your command performs a time-consuming task, where you can't respond to the interaction in less than 3 seconds, you can defer the interaction response (which will cause your bot to go into status thinking) and then send a followup message using await interaction.followup.send(). async def graph(self, interaction: discord.Interaction, symbol): await interaction.response.defer() # put this on the start video_path = "stock_graph.mp4" imageio.mimsave(video_path, ani_frames, fps=8) # maybe this task is taking long? asyncio.sleep(10) # why are you sleeping 10 seconds? await interaction.followup.send(file=discord.File(video_path)) plt.close()
Question: The code below used to work in Pydantic V1: from pydantic import BaseModel class CustomInt(int): """Custom int.""" pass class CustomModel(BaseModel): """Custom model.""" custom_int: CustomInt  Using it with Pydantic V2 will trigger an error. The error can be resolved by including arbitrary_types_allowed=True in model_config, but is there a better solution? Kindly note that the docs suggest using Annotated, but that doesn't allow defining custom docstring, which is desirable in my use case.

Answer: You can keep using a class which inherits from a type by defining core schema on the class: from typing_extensions import Any from pydantic import GetCoreSchemaHandler, TypeAdapter from pydantic_core import CoreSchema, core_schema class CustomInt(int): """Custom int.""" ... @classmethod def __get_pydantic_core_schema__( cls, source_type: Any, handler: GetCoreSchemaHandler ) -> CoreSchema: return core_schema.no_info_after_validator_function(cls, handler(int))   Note: These are advanced methods and may have undesirable side effects. Further documentation here: https://docs.pydantic.dev/latest/api/pydantic_core/
Question: I want to execute this code in google colab but I get following error: from llama_index.prompts.prompts import SimpleInputPrompt # Create a system prompt system_prompt = """
 <> more string here.<> """ query_wrapper_prompt = SimpleInputPrompt("{query_str} 
")  Error: /usr/local/lib/python3.10/dist-packages/pydantic/_internal/_config.py:269: UserWarning: Valid config keys have changed in V2: * 'allow_population_by_field_name' has been renamed to 'populate_by_name' warnings.warn(message, UserWarning) --------------------------------------------------------------------------- PydanticUserError Traceback (most recent call last) <ipython-input-36-c45796b371fe> in <cell line: 3>() 1 # Import the prompt wrapper... 2 # but for llama index ----> 3 from llama_index.prompts.prompts import SimpleInputPrompt 4 # Create a system prompt 5 system_prompt = """
 <> 6 frames /usr/local/lib/python3.10/dist-packages/pydantic/deprecated/class_validators.py in root_validator(pre, skip_on_failure, allow_reuse, *__args) 226 mode: Literal['before', 'after'] = 'before' if pre is True else 'after' 227 if pre is False and skip_on_failure is not True: --> 228 raise PydanticUserError( 229 'If you use `@root_validator` with pre=False (the default) you MUST specify `skip_on_failure=True`.' 230 ' Note that `@root_validator` is deprecated and should be replaced with `@model_validator`.', PydanticUserError: If you use `@root_validator` with pre=False (the default) you MUST specify `skip_on_failure=True`. Note that `@root_validator` is deprecated and should be replaced with `@model_validator`. For further information visit https://errors.pydantic.dev/2.1.1/u/root-validator-pre-skip  If I follow the link, there is no solution for my case. How can I solve that problem? Thanks in forward.

Answer: In my env, I have pip list | grep pydantic pydantic 2.2.1  I fix the problem, by downgrading pydantic version pip install pydantic==1.10.9
Question: I have a co-occurence matrix in pyspark for co-ocurrences of certain keywords A,B,C  A B C A 5 1 0 B 1 3 2 C 0 2 3  How can I calculate the jaccard similarity from this matrix in Python for all keywords. Is there any library available to do that or should I simply compute the similarity by using the Jaccard similarity formula?

Answer: Let's assume the co-occurrence matrix is given as a list of lists: com = [[5, 1, 0], [1, 3, 2], [0, 2, 3]] n_elem = len(com)  The Jaccard similarity of two sets A and B is given by |A ∩ B| / |A ∪ B|. The co-occurrence matrix gives the value of |A|, |B|, and |A ∩ B|. The value of |A ∪ B| is simply |A| + |B| - |A ∩ B|, which we can find the Jaccard index. First, let's create a list of lists containing ones that is the same size as com. The default value is 1 because the similarity index of a set with itself is 1, and we will not calculate these elements: similarity = [[1 for _ in row] for row in com]  Now, we can loop over each pair of values in com and calculate the similarities. The inner loop starts at i+1 because similarity[i][j] is identical to similarity[j][i], so we only need to calculate the upper triangle of the matrix: for i in range(n_elem): a = com[i][i] # |A| for j in range(i+1, n_elem): b = com[j][j] # |B| aib = com[i][j] # |A ∩ B| aub = a + b - aib # |A ∪ B| # Set both off-diagonal elements simultaneously similarity[i][j] = similarity[j][i] = aib / aub  This leaves us with the following similarity matrix: [[1 , 0.14285714285714285, 0.0], [0.14285714285714285, 1 , 0.5], [0.0 , 0.5 , 1]]   Now, if your co-occurrence matrix is a numpy array (or you're open to using numpy), you can speed up this computation by outsourcing the loops to numpy's C backend. import numpy as np com_arr = np.array([[5, 1, 0], [1, 3, 2], [0, 2, 3]]) n_elem = com_arr.size  First, we can get the occurrence of each element using the diagonal of the matrix: occ = np.diag(com_arr) # array([5, 3, 3])  Next, create the matrix of |A ∪ B|. Remember that |A ∩ B| is already specified by com_arr: aub = occ[:, None] + occ[None, :] - com_arr  Since occ is a 1-d array, adding a None index will create a 2-d array of one column (a column vector of shape (3, 1)) and one row (a row vector of shape (1, 3)) respectively. When adding a row vector to a column vector, numpy automatically broadcasts the dimensions so that you end up with a (in this case) square matrix of shape (3, 3). Now, aub looks like this: array([[5, 7, 8], [7, 3, 4], [8, 4, 3]])  Finally, divide the intersection by the union: similarity = com_arr / aub  et voila, we have the same values as before: array([[1. , 0.14285714, 0. ], [0.14285714, 1. , 0.5 ], [0. , 0.5 , 1. ]])
Question: I am trying to sort the content of a text file. Currently, I'm sorting it using excel. I wanted to use python script to automatically sort the contents. The code below does not produce the needed output. Any idea will be very much appreciated.  def sorting(filename): infile = open(filename) words = [] for line in infile: temp = line.split() for i in temp: words.append(i) infile.close() words.sort() outfile = open("result.txt", "w") for i in words: outfile.writelines(i) outfile.writelines(" ") outfile.close() sorting("myfile.txt")  Current code output:  -0.162 -0.638 -2.248 -2.348 -3.427 -4.622 0.00 0.162 0.176 -> more data...  myfile.txt # file needed to be sorted  ABC1234 -2.385 05:58 1234DCE -3.430 05:58 98ACD12 12.574 05:58 18DDD12 11.564 05:58 453FGH2A 0.351 05:58 A2DA21 0.00 05:58 8FF3EFA -0.720 05:58 ABC1234 -2.348 06:11 1234DCE -3.427 06:11 98ACD12 11.883 06:11 18DDD12 10.883 06:11 453FGH2A 0.176 06:11 A2DA21 0.162 06:11 8FF3EFA 0.888 06:11 ABC1234 -2.248 06:18 1234DCE -4.622 06:18 98ACD12 13.356 06:18 18DDD12 10.915 06:18 453FGH2A 0.00 06:18 A2DA21 -0.162 06:18 8FF3EFA -0.638 06:18  Intended Output: # to be sorted based on the latest time.  Name 06:18 06:11 05:58 ABC1234 -2.248 -2.348 -2.385 1234DCE -4.622 -3.427 -3.430 98ACD12 13.356 11.883 12.574 18DDD12 10.915 10.883 11.564 453FGH2A 0.00 0.176 0.351 A2DA21 -0.162 0.162 0.00 8FF3EFA -0.638 0.888 -0.720 

Answer: I don't usually argue for pandas as the first solution, but in this case I think it is the right answer: import pandas as pd data = {} for row in open('x.csv'): cols = row.rstrip().split() if cols[2] not in data: data[cols[2]] = {} data[cols[2]][cols[0]] = cols[1] print(data) df = pd.DataFrame(data) print(df)  Output: {'05:58': {'ABC1234': '-2.385', '1234DCE': '-3.430', '98ACD12': '12.574', '18DDD12': '11.564', '453FGH2A': '0.351', 'A2DA21': '0.00', '8FF3EFA': '-0.720'}, '06:11': {'ABC1234': '-2.348', '1234DCE': '-3.427', '98ACD12': '11.883', '18DDD12': '10.883', '453FGH2A': '0.176', 'A2DA21': '0.162', '8FF3EFA': '0.888'}, '06:18': {'ABC1234': '-2.248', '1234DCE': '-4.622', '98ACD12': '13.356', '18DDD12': '10.915', '453FGH2A': '0.00', 'A2DA21': '-0.162', '8FF3EFA': '-0.638'}} 05:58 06:11 06:18 ABC1234 -2.385 -2.348 -2.248 1234DCE -3.430 -3.427 -4.622 98ACD12 12.574 11.883 13.356 18DDD12 11.564 10.883 10.915 453FGH2A 0.351 0.176 0.00 A2DA21 0.00 0.162 -0.162 8FF3EFA -0.720 0.888 -0.638  (Side note -- you could eliminate two lines of code by using defaultdict for data. In this case, I don't think it's worth the trouble.)