eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a table with rows in this format where the integers are a count:  A B C D E 0 a 2 0 3 x 1 b 1 2 0 y  I'd like to convert it into a format where each count is a one hot encoded row:  A B C D E 0 a 1 0 0 x 1 a 1 0 0 x 2 a 0 0 1 x 3 a 0 0 1 x 4 a 0 0 1 x 5 b 1 0 0 y 6 b 0 1 0 y 7 b 0 1 0 y  I wrote inefficient code which achieves this # Sample DataFrame data = { 'A': ['a', 'b'], 'B': [2, 1], 'C': [0, 2], 'D': [3, 0], 'E': ['x', 'y'] } df = pd.DataFrame(data) new_df = pd.DataFrame(columns=df.columns) for index, row in df.iterrows(): first_val = row.iloc[0] last_val = row.iloc[-1] middle_vals = row.iloc[1:-1].astype(int) for i in range(len(middle_vals)): new_data = [first_val] + [1 if i == j else 0 for j in range(len(middle_vals))] + [last_val] new_rows = pd.DataFrame([new_data] * middle_vals.iloc[i], columns=df.columns) new_df = pd.concat([new_df, new_rows], ignore_index=True)  Any tips for vectorizing this operation which is incredibly slow? I realize a concat operation per iteration is a big issue, so I did try a batching solution where I collect chunks of new_rows and then concat. This remains slow.

Answer: Here is a full numpy solution, I would expect this to be faster than reshaping: import numpy as np num_cols = ['B', 'C', 'D'] # convert to numpy array a = df[num_cols].to_numpy() # build indices to repeat idx = np.repeat(np.arange(a.shape[0]), a.sum(1)) # array([0, 0, 0, 0, 0, 1, 1, 1]) # build column indices to repeat cols = np.repeat(np.tile(np.arange(a.shape[1]), a.shape[0]), a.flat) # array([0, 0, 2, 2, 2, 0, 1, 1]) # assign 1s b = np.zeros((len(idx), len(num_cols)), dtype=int) b[np.arange(len(idx)), cols] = 1 # repeat and update DataFrame out = df.iloc[idx] out.loc[:, num_cols] = b  Output:  A B C D E 0 a 1 0 0 x 0 a 1 0 0 x 0 a 0 0 1 x 0 a 0 0 1 x 0 a 0 0 1 x 1 b 1 0 0 y 1 b 0 1 0 y 1 b 0 1 0 y  timings # setup N = 10_000 df = (pd.DataFrame(np.clip(np.random.randint(-10, 15, (N, 10)), 0, 15)).assign(A=np.arange(N), E=np.arange(N)) [['A', 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 'E']] ) # numpy approach (this one) 45.2 ms ± 6.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) # list approach (@EmiOB) 249 ms ± 36.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # pure pandas melt+repeat+pivot approach 3.68 s ± 146 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  comparison for up to ~2M rows (10 cols of numbers)
Question: I'm new to Equinox and JAX but wanted to use them to simulate a dynamical system. But when I pass my system model as an Equinox module to jax.lax.scan I get the unhashable type error in the title. I understand that jax expects the function argument to be a pure function but I thought an Equinox Module would emulate that. Here is a test script to reproduce the error import equinox as eqx import jax import jax.numpy as jnp class EqxModel(eqx.Module): A: jax.Array B: jax.Array C: jax.Array D: jax.Array def __call__(self, states, inputs): x = states.reshape(-1, 1) u = inputs.reshape(-1, 1) x_next = self.A @ x + self.B @ u y = self.C @ x + self.D @ u return x_next.reshape(-1), y.reshape(-1) def simulate(model, inputs, x0): xk = x0 outputs = [] for uk in inputs: xk, yk = model(xk, uk) outputs.append(yk) outputs = jnp.stack(outputs) return xk, outputs A = jnp.array([[0.7, 1.0], [0.0, 1.0]]) B = jnp.array([[0.0], [1.0]]) C = jnp.array([[0.3, 0.0]]) D = jnp.array([[0.0]]) model = EqxModel(A, B, C, D) # Test simulation inputs = jnp.array([[0.0], [1.0], [1.0], [1.0]]) x0 = jnp.zeros(2) xk, outputs = simulate(model, inputs, x0) assert jnp.allclose(xk, jnp.array([2.7, 3.0])) assert jnp.allclose(outputs, jnp.array([[0.0], [0.0], [0.0], [0.3]])) # This raises TypeError xk, outputs = jax.lax.scan(model, x0, inputs)  What is unhashable type: 'ArrayImpl' referring to? Is it the arrays A, B, C, and D? In this model, these matrices are parameters and therefore should be static for the duration of the simulation. I just found this issue thread that might be related:  lax.scan for equinox Modules 

Answer: Owen Lockwood (lockwo) has provided an explanation and answer in this issue thread, which I will re-iterate below.  I believe your issue is happening because jax tries to hash the function you are scanning over, but it can't hash the arrays that are in the module. There are probably a number of things that you could do to solve this, the simplest being to just curry the model, e.g. xk, outputs = jax.lax.scan(lambda carry, y: model(carry, y), x0, inputs) works fine  Or, re-written in terms of the variable names I am using: xk, outputs = jax.lax.scan(lambda xk, uk: model(xk, uk), x0, inputs)
Question: I have a list of phrases and I need to be able to identify whether each row in a dataset contains all the words from any of the phrases in my list. Take my example problem below. I have a dataset where the column "Search" contains some browser searches. I also have a list called "phrases" that contains the phrases I'm trying to find within the Search column. import pandas as pd import numpy as np text = [('how to screenshot on mac', 0), ('how to take screenshot?', 0), ('how to take screenshot on windows', 0), ('when is christmas', 0), ('how many days until christmas', 0), ('how many weeks until christmas', 0), ('how much is the new google pixel 8', 0), ('which google pixel versions are available', 0), ('how do I do google search on my pixel phone 7a', 0)] labels = ['Search','Random_Column'] df = pd.DataFrame.from_records(text, columns=labels) phrases = ['mac screenshot', 'days until christmas', 'google pixel 7a']  I don't care about the order of the words within "phrases" and there can be other before before, within, and after the phrase, but I need to make sure that only the df rows that contain all the words within any of the phrases are identified. Therefore, the expected output would be like this:  Search Random_Column Match 0 how to screenshot on mac 0 True 1 how to take screenshot? 0 False 2 how to take screenshot on windows 0 False 3 when is christmas 0 False 4 how many days until christmas 0 True 5 how many weeks until christmas 0 False 6 how much is the new google pixel 8 0 False 7 which google pixel versions are available 0 False 8 how do I do google search on my pixel phone 7a 0 True  I have found a lot of solutions for instances where the "phrases" list is made up of single words (e.g. here, here, and here) but I'm struggling to find a solution where I need to match full phrases. I also tried to implement this solution but could not get it to work for a dataset.

Answer: You have to loop over all phrase until you find a match. An efficient option would be to use sets (set.issubset) combined with any: # convert the phrases to set sets = [set(s.split()) for s in phrases] # [{'screenshot', 'mac'}, {'until', 'christmas', 'days'}, # {'google', 'pixel', '7a'}] # for each string, check if one of the sets is a subset # if a match is found, return True immediately df['Match'] = [any(S.issubset(lst) for S in sets) for lst in map(str.split, df['Search'])]  Output:  Search Random_Column Match 0 how to screenshot on mac 0 True 1 how to take screenshot? 0 False 2 how to take screenshot on windows 0 False 3 when is christmas 0 False 4 how many days until christmas 0 True 5 how many weeks until christmas 0 False 6 how much is the new google pixel 8 0 False 7 which google pixel versions are available 0 False 8 how do I do google search on my pixel phone 7a 0 True
Question: When trying to construct an example where a == b is not the same as b == a, it seems that I have accidentally constructed an example where a == b is not the same as a.__eq__(b): class A: def __eq__(self, other: object) -> bool: return type(other) is A class B(A): pass if __name__ == '__main__': a = A() b = B() assert not a.__eq__(b) # as expected assert not (a == b) # Why does this fail?  Can somebody explain to me why the last assertion fails? I expected it to be the same as the second one.

Answer: The relevant quote explaining what happens is located in the documentation:  If the operands are of different types, and the right operand’s type is a direct or indirect subclass of the left operand’s type, the reflected method of the right operand has priority, otherwise the left operand’s method has priority. Virtual subclassing is not considered.
Question: I tried to replace the some of major values in the tensor with the unique value while maintaining at least one value. For example, given edge_index, I want to change it as below. edge_index = torch.as_tensor([[0, 0, 1, 2, 3, 4, 6, 7, 7, 8], [1, 2, 2, 4, 4, 5, 0, 1, 3, 7]]) result = torch.as_tensor([[5, 0, 1, 2, 3, 8, 6, 7, 7, 8], [1, 2, 2, 4, 4, 5, 0, 1, 3, 6]])  In detail, some values appeared more than 2 times in edge_index (i.e., 0, 1, 2, 4, 7). To avoid unique values, I need to change some of them with exceptional values (i.e., 5, 6, 8). I tried to do it as below, but my code couldn't ensure the condition; all values need to appear at least 2 times. # Counts values n_id, n_counts = torch.unique(edge_index, return_counts=True) unique_nodes = n_id[n_counts==1] #tensor([5, 6, 8]) major_nodes = n_id[n_counts>2] #tensor([0, 1, 2, 4, 7]) # Find the index where the major_nodes located major_s_idx = (edge_index[..., None] == major_nodes).any(-1)[0].nonzero()[:, 0] #tensor([0, 1, 2, 3, 5, 7, 8]) major_t_idx = (edge_index[..., None] == major_nodes).any(-1)[1].nonzero()[:, 0] #tensor([0, 1, 2, 3, 4, 6, 7, 9]) result = edge_index.clone() result[0][major_s_idx[torch.randperm(len(major_s_idx))[:len(unique_nodes)]]] = unique_nodes result[1][major_t_idx[torch.randperm(len(major_t_idx))[:len(unique_nodes)]]] = unique_nodes result # tensor([[0, 0, 6, 2, 3, 4, 6, 5, 8, 8], # [5, 2, 2, 4, 6, 5, 0, 8, 3, 7]]) # 1 is disappeared and 7 remained only one time  Note that the correct result does not need to be the same as the result in the first code block. Just reference.

Answer: Your issue arises because you're replacing some of the major nodes with unique nodes randomly, without checking if the major node still appears elsewhere. Create a list of nodes that appear twice, then for each node find all its occurrences. Finally, randomly select one occurrence to keep and replace the rest with unique nodes: import torch edge_index = torch.tensor([[0, 0, 1, 2, 3, 4, 6, 7, 7, 8], [1, 2, 2, 4, 4, 5, 0, 1, 3, 7]]) n_id, n_counts = torch.unique(edge_index, return_counts=True) unique_nodes = n_id[n_counts==1] major_nodes = n_id[n_counts>2] result = edge_index.clone() for major_node in major_nodes: occurrences = torch.nonzero(result == major_node, as_tuple=True) keep_idx = torch.randint(len(occurrences[0]), (1,))[0] for i in range(len(occurrences[0])): if i != keep_idx and len(unique_nodes) > 0: result[occurrences[0][i], occurrences[1][i]] = unique_nodes[0] unique_nodes = unique_nodes[1:] print(result)
Question: We were trying to make calls in parallel to LLMs hosted in Bedrock, from a lambda layer (in python) only to discover that boto3 does not support async. Is there any workaround? I am looking into aiobotocore / aioboto3, but I do not find any example with Bedrock. Any hint appreciated and thank you very much! This is a minimal sample of the code I intended to use, but runs in sequence instead of parallel: nest_asyncio.apply() # async summaries async def _into_comment(segments: list[str]): bedrock = boto3.client( service_name="bedrock-runtime", aws_access_key_id=aws_access_key, aws_secret_access_key=aws_secret_key, aws_session_token=aws_session_token, region_name=aws_region ) async def sum_up(segment: str): body = json.dumps({ "max_tokens": 256, "messages": [{"role": "user", "content": f"Sumarize this: {segment}"}], "anthropic_version": "bedrock-2023-05-31" }) return bedrock.invoke_model(body=body, modelId=model_id) summaries = await asyncio.gather(*[sum_up(segment) for segment in segments]) return summaries summaries = asyncio.run(_into_comment(segments)) 

Answer: If you are using Anthropic, you can use the AsyncAnthropicBedrock API. from anthropic import AsyncAnthropicBedrock model_id = "my_model_id" user_message = "Hello Claude!" client = AsyncAnthropicBedrock() message = await client.messages.create( model=model_id, max_tokens=1024, messages=[ {"role": "user", "content": user_message} ] )
Question: I am trying to filter a dataframe to find the first occurrence of a maximum value over a category column. In my data there is no guarantee that there is a single unique maximum value, there could be multiple values, but i only need the first occurance. Yet I can't seem to find a way to limit the max part of the filter, currently I am then adding a further filter on another column generally a time based one and taking the minimum value. df = pl.DataFrame( { "cat": [1, 1, 1, 2, 2, 2, 2, 3, 3, 3], "max_col": [12, 24, 36, 15, 50, 50, 45, 20, 40, 60], "other_col": [25, 50, 75, 125, 150, 175, 200, 225, 250, 275], } ) df = df.filter(pl.col("max_col") == pl.col("max_col").max().over("cat")).filter( pl.col("other_col") == pl.col("other_col").min().over("cat") ) shape: (3, 3) ┌─────┬─────────┬───────────┐ │ cat ┆ max_col ┆ other_col │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════════╪═══════════╡ │ 1 ┆ 36 ┆ 75 │ │ 2 ┆ 50 ┆ 150 │ │ 3 ┆ 60 ┆ 275 │ └─────┴─────────┴───────────┘  However, I'd prefer to simplify the above to only require passing in references to the max and category columns. Am I missing something obvious here? EDIT: Added example dataframe and output.

Answer: You can add .is_first_distinct() to the filter to keep only the first max. df.filter( pl.all_horizontal( pl.col("max_col") == pl.col("max_col").max(), pl.col("max_col").is_first_distinct() ) .over("cat") )  shape: (3, 3) ┌─────┬─────────┬───────────┐ │ cat ┆ max_col ┆ other_col │ │ --- ┆ --- ┆ --- │ │ i64 ┆ i64 ┆ i64 │ ╞═════╪═════════╪═══════════╡ │ 1 ┆ 36 ┆ 75 │ │ 2 ┆ 50 ┆ 150 │ │ 3 ┆ 60 ┆ 275 │ └─────┴─────────┴───────────┘
Question: I have following code from typing import TypeVar, Type, overload T = TypeVar('T') @overload def foo(bar: Type[T]) -> T: ... @overload def foo(bar: Type[T] | None) -> T | None: ... def foo(bar: Type[T] | None) -> T | None: # implementation goes here ... class Bar: ... bar = foo(Bar) bar2 = foo(Bar | None) # No overload variant of "foo" matches argument type "UnionType"  How to properly type hint case for bar2?  I tried some others: Type[T | None], mypy says Overloaded function signature 2 will never be matched: signature 1's parameter type(s) are the same or broader removing 2nd overload (resulting in only Type[T] allowed), mypy says No overload variant of "foo" matches argument type "UnionType" (meaning 2nd overload is incorrect for that case anyways)

Answer: Only concrete classes are assignable to type[T]:  [...] the actual argument passed in at runtime must [...] be a concrete class object [...] — Special types in annotations § type[] | Python typing spec  (Admittedly, the spec isn't completely clear about this. However, the following paragraph holds true.) It is thus understood that an object of this type can be invoked to retrieve an instance of type T. type[T] | None means such a thing, or None; thus, bar might or might not be invocable. def foo(bar: type[T] | None) -> None: if bar is not None: reveal_type(bar) # type[T] instance = bar() # fine  However, Bar | None returns an instance of UnionType at runtime, and objects of this kind cannot be invoked. foo(Bar | None) # (Bar | None) is not None # bar() => error  What you want is the proposed TypeForm of PEP 747 (a draft PEP): def foo[T](bar: TypeForm[T]) -> T: ... reveal_type(foo(Bar | None)) # Bar | None
Question: I have a python script and trying to add type hints to the code, Following is the sample code (without type hints the code works) using mypy. values_int: list[int] = [1, 2, 3, 4] values_str: list[str] = ["1", "2", "3", "4"] def bar(*, x: list[int]) -> bool: # processing return True def baz(*, y: list[str]) -> bool: # processing return True def foo(*, values: list[int] | list[str]) -> bool: status: bool = False if isinstance(values[0], int): x: list[int] = values status = bar(x=x) elif isinstance(values[0], str): # case-1 # status = baz(y=values) # case-2 y: list[str] = values status = baz(y=y) return status foo(values=values_str)  Errors for: # case-1 # error: Argument "y" to "baz" has incompatible type "list[int] | list[str]"; expected "list[str]" # case-2 # error: Incompatible types in assignment (expression has type "list[int] | list[str]", variable has type "list[str]") 

Answer: isinstance(a[b], ...) is not (yet) supported as a type narrowing construct. For what it's worth, Pyright also doesn't support it. Perhaps you want a custom TypeIs: (playgrounds: Mypy, Pyright) from typing_extensions import TypeIs def is_list_of_ints(v: list[Any]) -> TypeIs[list[int]]: return isinstance(v[0], int) def is_list_of_strs(v: list[Any]) -> TypeIs[list[str]]: return isinstance(v[0], int)  def foo(*, values: list[int] | list[str]) -> None: reveal_type(values) # list[int] | list[str] if is_list_of_ints(values): reveal_type(values) # list[int] bar(x=values) # fine elif is_list_of_strs(values): reveal_type(values) # list[str] baz(y=values) # fine  Note that Mypy 1.10.0 has a bug with TypeIs: it incorrectly determines the second conditional branch to be unreachable: def foo(*, values: list[int] | list[str]) -> None: reveal_type(values) # list[int] | list[str] if is_list_of_ints(values): reveal_type(values) # list[int] bar(x=values) # fine elif is_list_of_strs(values): reveal_type(values) # error: Statement is unreachable baz(y=values)  A workaround is to use TypeGuard instead: (playground) from typing import TypeGuard def is_list_of_ints(v: list[Any]) -> TypeGuard[list[int]]: return isinstance(v[0], int) def is_list_of_strs(v: list[Any]) -> TypeGuard[list[str]]: return isinstance(v[0], int)  The difference between TypeIs and TypeGuard is explained in PEP 742.
Question:  I'm relatively new to platforms like vscode, and I seem to run into problems whenever I try to run a python program without debugging (Ctrl + F5). A window pops up saying "Could not find debugpy path" and gives me the option of either opening launch.json or cancelling. Running from the "run file" button on the right works, so I'm just curious what is causing this problem. I opened launch.json and it's mostly empty: { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [] }  

Answer: Turns out I needed to download a Python debugger in the VSCode extensions. I used Python Debugger from Microsoft.
Question: I get:  return r.group() if r := re.match(rx,l) else None ^  SyntaxError: invalid syntax whereas  return r.group() if (r := re.match(rx,l)) else None  is accepted. What is invalid about the first's syntax? And what other interpretation of it is there, than the second, such that it is not unambiguous?

Answer: Limiting the valid places in which the walrus operator ("assignment expression" for those who have killed their inner child) can be used was a mostly elective thing. They didn't want it replacing = for assignment, or getting used constantly to condense multiple lines of code down to one for the fun of it, so they put limitations on its syntax that make it awkward outside of the cases it was really designed for. The biggest limitation was that they require parentheses in cases they didn't strictly need to (e.g. the most obvious case is that the language could trivially have implemented it such that x := 1 by itself worked, but chose not to do so). The language's grammar spec describes the precise rules (emphasis added):  Assignment expressions must be surrounded by parentheses when used as expression statements and when used as sub-expressions in slicing, conditional, lambda, keyword-argument, and comprehension-if expressions and in assert, with, and assignment statements. In all other places where they can be used, parentheses are not required, including in if and while statements.  Basically, unparenthesized walruses are definitionally incompatible with conditional expressions, because the Python language designers said so.
Question: Suppose I have a vector of parameters p which parameterizes a set of matrices A_1(p), A_2(p),...,A_N(p). I have a computation in which for some list of indices q of length M, I have to compute A_{q_M} * ... * A_{q_2} * A_{q_1} * v for several different q s. Each q has a different length, but crucially doesn't change! What changes, and what I wish to take gradients against is p. I'm trying to figure out how to convert this to performant JAX. One way to do it is to have some large matrix Q which contains all the different qs on each row, padded out with identity matrices such that each multiplication chain is the same length, and then scan over a function that switch es between N different functions doing matrix-vector multiplications by A_n(p). However -- I don't particularly like the idea of this padding. Also, since Q here is fixed, is there potentially a smarter way to do this? The distribution of lengths of q s has a very long tail, so Q will be dominated by padding. EDIT: Here's a (edit 2: functional) minimal example sigma0 = jnp.eye(2) sigmax = jnp.array([[0, 1], [1, 0]]) sigmay = jnp.array([[0, -1j], [1j, 0]]) sigmaz = jnp.array([[1, 0], [0, -1]]) sigma = jnp.array([sigmax, sigmay, sigmaz]) def gates_func(params): theta = params["theta"] epsilon = params["epsilon"] n = jnp.array([jnp.cos(theta), 0, jnp.sin(theta)]) omega = jnp.pi / 2 * (1 + epsilon) X90 = expm(-1j * omega * jnp.einsum("i,ijk->jk", n, sigma) / 2) return { "Z90": expm(-1j * jnp.pi / 2 * sigmaz / 2), "X90": X90 } def multiply_out(params): gate_lists = [["X90", "X90"], ["X90","Z90"], ["Z90", "X90"], ["X90","Z90","X90"]] gates = gates_func(params) out = jnp.zeros(len(gate_lists)) for i, gate_list in enumerate(gate_lists): init = jnp.array([1.0,0.0], dtype=jnp.complex128) for g in gate_list: init = gates[g] @ init out = out.at[i].set(jnp.abs(init[0])) return out params = dict(theta=-0.0, epsilon=0.001) multiply_out(params) 

Answer: The main issue here is that JAX does not support string inputs. But you can use NumPy to manipulate string arrays and turn them into integer categorical arrays that can then be used by jax.jit and jax.vmap. The solution might look something like this: import numpy as np def gates_func_int(params, gate_list_vals): g = gates_func(params) identity = jnp.eye(*list(g.values())[0].shape) return jnp.stack([g.get(val, identity) for val in gate_list_vals]) @jax.jit def multiply_out_2(params): # compile-time pre-processing gate_lists = [["X90", "X90"], ["X90","Z90"], ["Z90", "X90"], ["X90","Z90","X90"]] max_size = max(map(len, gate_lists)) gate_array = np.array([gates + [''] * (max_size - len(gates)) for gates in gate_lists]) gate_list_vals, gate_list_ints = np.unique(gate_array, return_inverse=True) gate_list_ints = gate_list_ints.reshape(gate_array.shape) # runtime computation gates = gates_func_int(params, gate_list_vals)[gate_list_ints] initial = jnp.array([[1.0],[0.0]], dtype=jnp.complex128) return jax.vmap(lambda g: jnp.abs(jnp.linalg.multi_dot([*g, initial]))[0])(gates).ravel() multiply_out_2(params)