eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have code that aims to generate a graph from an adjacency matrix from a table correlating workers with their manager. The source is a table with two columns (Worker, manager). It still works perfectly from a small mock data set, but fails unexpectedly with the real data: import pandas as pd import networkx as nx # Read input df = pd.read_csv("org.csv") # Create the input adjacency matrix am = pd.DataFrame(0, columns=df["Worker"], index=df["Worker"]) # This way, it is impossible that the dataframe is not square, # or that index and columns don't match # Fill the matrix for ix, row in df.iterrows(): am.at[row["manager"], row["Worker"]] = 1 # At this point, am.shape returns a square dataframe (2825,2825) # Generate the graph G = nx.from_pandas_adjacency(am, create_using=nx.DiGraph)  This returns: NetworkXError: Adjacency matrix not square: nx,ny=(2825, 2829) And indeed, the dimensions reported in the error are not the same as in those of the input dataframe am. Does anyone have an idea of what happens in from_pandas_adjacency that could lead to this mismatch?

Answer: In: am = pd.DataFrame(0, columns=df["Worker"], index=df["Worker"]) # This way, it is impossible that the dataframe is not square,  your DataFrame is indeed square, but when you later assign values in the loop, if you have a manager that is not in "Worker", this will create a new row: am.at[row["manager"], row["Worker"]]  Better avoid the loop, use a crosstab, then reindex on the whole set of nodes: am = pd.crosstab(df['manager'], df['Worker']) nodes = am.index.union(am.columns) am = am.reindex(index=nodes, columns=nodes, fill_value=0)  Even better, if you don't really need the adjacency matrix, directly create the graph with nx.from_pandas_edgelist: G = nx.from_pandas_edgelist(df, source='manager', target='Worker', create_using=nx.DiGraph)  Example: # input df = pd.DataFrame({'manager': ['A', 'B', 'A'], 'Worker': ['D', 'E', 'F']}) # adjacency matrix A B D E F A 0 0 1 0 1 B 0 0 0 1 0 D 0 0 0 0 0 E 0 0 0 0 0 F 0 0 0 0 0 # adjacency matrix with your code Worker D E F Worker D 0.0 0.0 0.0 E 0.0 0.0 0.0 F 0.0 0.0 0.0 A 1.0 NaN 1.0 # those rows are created B NaN 1.0 NaN # after initializing am  Graph:
Question: Solved! - Please check the answer. I wrote a library where headers and python bindings are auto-generated. For example dummy_bind.cpp for dummy_message.h and each _bind.cpp file has PYBIND11_MODULE call in it for their specific class. There are dozens of other _bind.cpp files for other headers. What should be the module name for each file when calling the PYBIND11_MODULE like: PYBIND11_MODULE(protocol_name, m) { /// … }  If I use protocol_name in each PYBIND11_MODULE(protocol_name, m) call, when compiling I get multiple definition error like: multiple definition of PyInit_protocol_name. If I generate special module name for each message like PYBIND11_MODULE(protocol_name_dummy, m) the extension is compiled but I think I need to import each module one by one which is not viable. Should I do all exports inside a single PYBIND11_MODULE call? Thanks in advance.

Answer: I've actually solved this by generating proxy functions in _bind.cpp files. For instance, in message1_bind.cpp I've defined a function void init_message1(pyinit11::module& m) and then in main_bind.cpp I call them all inside PYBIND11_MODULE(protocol_name, m) so I only have only one PYBIND11_MODULE() call. Here's a minimal example to describe the method better: message1_bind.cpp: #include <pybind11/pybind11.h> #include "messages/message_1.h" void init_message1(pybind11::module& m) { pybind11::class_<protocol_name::Message1>(m, "Message1").def(pybind11::init<>()); m.def("serialize_message1", &protocol_name::serialize_message_1); m.def("deserialize_message1", &protocol_name::deserialize_message_1); }  And then inside main_bind.cpp #include <pybind11/pybind11.h> #include "message1_bind.cpp" /// ... all other includes PYBIND11_MODULE(protocol_name, m) { init_message1(m); /// init_message2(m) and so on.. }
Question: Say I have an array as follows: arr = np.asarray([1, 2, 3, 4294967296, 100], dtype=np.uint64)  I now want two arrays, one array with the lower 32 bits of every element, and one with the upper 32 bits of every element, preferably by using views and minimizing copies, to get something like this: upper = np.array([0, 0, 0, 1, 0], dtype=np.uint32) lower = np.array([1, 2, 3, 0, 100], dtype=np.uint32)  I tried the following: lower = arr.view() & 0xFFFFFFFF upper = np.bitwise_right_shift(arr.view(), 32)  But this results in a copy for the upper bits due to the bitshift, and both arrays are still of type uint64. Are there further optimizations I can try or am I out of luck and need to eat up the extra copies?

Answer: You can use structured arrays to split uint64 into two uint32 views without copying: # Create a structured view of the array (assuming little-endian system) view = arr.view(dtype=np.dtype([('lower', np.uint32), ('upper', np.uint32)])) # Extract views lower = view['lower'] upper = view['upper']  This creates memory views not copies, and preserves the uint32 dtypes. Alternative using views: This alternative also creates views without copying data, but my benchmarks show it can be significantly faster than the structured dtype approach. # View the uint64 array as uint32 (each uint64 becomes two uint32) arr_u32 = arr.view(np.uint32) # Extract the lower and upper 32 bits # For little-endian systems, index 0 is lower bits and index 1 is upper bits lower, upper = arr_u32[..., 0], arr_u32[..., 1]
Question: I am writing a simple project in python. My version of python is: 3.13.1 . I am using IntelliJ and Python plugin with version: 241.18034.62. I would like to debug my project but when I try to debug I am getting many errors: AttributeError: '_MainThread' object has no attribute 'isAlive'. Did you mean: 'is_alive'?  bigger part of stacktrace: C:\projects\mat\venv\Scripts\python.exe -X pycache_prefix=C:\Users\mylogin\AppData\Local\JetBrains\IntelliJIdea2024.1\cpython-cache C:/Users/mylogin/AppData/Roaming/JetBrains/IntelliJIdea2024.1/plugins/python/helpers/pydev/pydevd.py --multiprocess --qt-support=pyqt5 --client 127.0.0.1 --port 4095 --file C:\projects\mat\org\aa\aa\mat\delivery_processor.py Connected to pydev debugger (build 241.18034.62) Traceback (most recent call last): File "C:\Users\mylogin\AppData\Roaming\JetBrains\IntelliJIdea2024.1\plugins\python\helpers\pydev\_pydevd_bundle\pydevd_pep_669_tracing.py", line 238, in py_start_callback if not is_thread_alive(thread): ~~~~~~~~~~~~~~~^^^^^^^^ File "C:\Users\mylogin\AppData\Roaming\JetBrains\IntelliJIdea2024.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_is_thread_alive.py", line 18, in is_thread_alive return t.isAlive() ^^^^^^^^^ AttributeError: 'WriterThread' object has no attribute 'isAlive'. Did you mean: 'is_alive'?  From what I have understood there is some mismatch with versions. I have tried to change the dbugger properties:  But none if these helped. How should I setup my IntelliJ environment to be able to debug? I would like to avoid downgrading python version.

Answer: Path in error message may suggest that it is mistake in plugin, not in your code. It looks like code for Python2 which has function isAlive(). At this moment you may try to fix it. Open C:\Users\mylogin\AppData\Roaming\JetBrains\IntelliJIdea2024.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_is_thread_alive.py  and change isAlive into is_alive and it may help in this place. But if it was created for Python 2 then it may have more problems in other places.  You could send this problem it author(s) of this plugin.
Question: I am trying to apply target encoding to categorical features using the category_encoders.TargetEncoder in Python. However, I keep getting the following error: AttributeError: 'numpy.ndarray' object has no attribute 'groupby'  from category_encoders import TargetEncoder from sklearn.model_selection import train_test_split # Features for target encoding encoding_cols = ['grade', 'sub_grade', 'home_ownership', 'verification_status', 'purpose', 'application_type', 'zipcode'] # Train-Test Split X_train_cv, X_test, y_train_cv, y_test = train_test_split(x, y, test_size=0.25, random_state=1) X_train, X_test_cv, y_train, y_test_cv = train_test_split(X_train_cv, y_train_cv, test_size=0.25, random_state=1) # Initialize the Target Encoder encoder = TargetEncoder() # Apply Target Encoding for i in encoding_cols: X_train[i] = encoder.fit_transform(X_train[i], y_train) # **Error occurs here** X_test_cv[i] = encoder.transform(X_test_cv[i]) X_test[i] = encoder.transform(X_test[i])  want to successfully apply target encoding to the categorical columns without encountering the 'numpy.ndarray' object has no attribute 'groupby' error.

Answer: This is interesting. I can reproduce your error. It is related to the dtype. To solve the issue you need to force a conversion using its list values and set the name and index explicitly. y_train = pd.Series(y_train.tolist(), name='loan_status', index=y_train.index) This will convert your initial dtype of CategoricalDtype(categories=[1, 0], ordered=False, categories_dtype=int64) to dtype('int64') So you last cell in the Colab is now: # Initialize TargetEncoder encoder = ce.TargetEncoder(cols=encoding_cols) # Here is the list conversion and back to series y_train = pd.Series(y_train.tolist(), index=y_train.index) # Fit and transform the training data X_train = encoder.fit_transform(X_train, y_train)  and this works fine.
Question: I have a weight vector: weight_vec = pl.Series("weights", [0.125, 0.0625, 0.03125])  And also a DataFrame containing up to m variables. For simplicity, we will only have two varaibles: df = pl.DataFrame( { "row_index": [0, 1, 2, 3, 4], "var1": [1, 2, 3, 4, 5], "var2": [6, 7, 8, 9, 10], } )  The size (number of observations) for these variables can be very large (tens of millions of rows). I would like to:  For each variable, and each observation x_i, where i is the row index [0,...,4], I want to transform the value of x_i to the sumproduct of all past n's x_i value (including the current value [x_i,...x_i+n-1]), and the weight vector. n is the length of the given weight vector and n varies for different weight vector definition. Numerically, the value of var1 at observation index 0 is the sumproduct of the values of all [x_0, x_1, x_2] and all the values of the weight vector. When the row index appraoches to and end (e.g., max index - row index + 1 < n) => all the values will be assigned None.  We can assume that the height of the DataFrame is always larger or equal to the length of the weight vector to result in at least one valid result.   The resulting DataFrame should look like this: shape: (5, 3) ┌───────────┬─────────┬─────────┐ │ row_index ┆ var1 ┆ var2 │ │ --- ┆ --- ┆ --- │ │ i64 ┆ f64 ┆ f64 │ ╞═══════════╪═════════╪═════════╡ │ 0 ┆ 0.34375 ┆ 1.4375 │ │ 1 ┆ 0.5625 ┆ 1.65625 │ │ 2 ┆ 0.78125 ┆ 1.875 │ │ 3 ┆ null ┆ null │ │ 4 ┆ null ┆ null │ └───────────┴─────────┴─────────┘  Numeric Caldulations:  x_0_var1: (0.125 * 1 + 0.0625 * 2 + 0.03125 * 3 = 0.34375) x_2_var2: (0.125 * 8 + 0.0625 * 9 + 0.03125 * 10 = 1.875)  I am looking for a memory efficient, vectorized Polars operation to achieve such results.

Answer: Here is a solution that uses rolling. import numpy as np weight_vec_len: int = weight_vec.len() period = f"{weight_vec_len}i" df.rolling("row_index", period=period, offset=f"-1i").agg( pl.col(r"^var\d$") .extend_constant(np.nan, weight_vec_len - pl.len()) .dot(weight_vec) .fill_nan(None) .name.keep() )  shape: (5, 3) ┌───────────┬─────────┬─────────┐ │ row_index ┆ var1 ┆ var2 │ │ --- ┆ --- ┆ --- │ │ i64 ┆ f64 ┆ f64 │ ╞═══════════╪═════════╪═════════╡ │ 0 ┆ 0.34375 ┆ 1.4375 │ │ 1 ┆ 0.5625 ┆ 1.65625 │ │ 2 ┆ 0.78125 ┆ 1.875 │ │ 3 ┆ null ┆ null │ │ 4 ┆ null ┆ null │ └───────────┴─────────┴─────────┘
Question: I am new to JAX and trying to use it with PennyLane and optax to optimize a simple quantum circuit. However, I noticed that my print statement inside the cost function does not execute in every iteration. Specifically, it prints only once at the beginning and then stops appearing. The quantum circuit itself does not make sense; I just wanted to simplify the example as much as possible. I believe the circuit is not actually relevant to the question, but it's included as an example. Here is my code: import pennylane as qml import jax import jax.numpy as jnp import optax jax.config.update("jax_enable_x64", True) device = qml.device("default.qubit", wires=1) @qml.qnode(device, interface='jax') def circuit(params): qml.RX(params, wires=0) return qml.expval(qml.PauliZ(0)) def cost(params): print('Evaluating') return circuit(params) # Define optimizer params = jnp.array(0.1) opt = optax.adam(learning_rate=0.1) opt_state = opt.init(params) # JIT the gradient function grad = jax.jit(jax.grad(cost)) for epoch in range(5): print(f'{epoch = }') grad_value = grad(params) updates, opt_state = opt.update(grad_value, opt_state) params = optax.apply_updates(params, updates)  Expected output: epoch = 0 Evaluating epoch = 1 Evaluating epoch = 2 Evaluating epoch = 3 Evaluating epoch = 4 Evaluating  Actual output: epoch = 0 Evaluating epoch = 1 Evaluating epoch = 2 epoch = 3 epoch = 4  Question: Why is the print statement inside cost not executed after the first iteration? Is JAX caching the function call or optimizing it in a way that skips execution? How can I ensure that cost is evaluated in every iteration?

Answer: When working with JAX it is important to understand the difference between "trace time" and "runtime". For JIT compilation JAX does an abstract evaluation of the function when it is called first. This is used to "trace" the computational graph of the function and then create a fully compiled replacement, which is cached and then invoked on the next calls ("runtime") of the function. Now, Python's print statements are only evaluated at trace time and not at runtime, because the code of the function has been effectively replaced by a compiled version. For the case of printing during runtime, JAX has a special jax.debug.print function, you can use: def cost(params): jax.debug.print('Evaluating') return circuit(params)  More on the jax.debug utilities: https://docs.jax.dev/en/latest/debugging/index.html And JIT compilation: https://docs.jax.dev/en/latest/jit-compilation.html
Question: I have this Streamlit app: import streamlit as st st.title("Simulator") tab_names = ["tab1", "tab2"] tab1, tab2= st.tabs(tab_names) @st.dialog("Edit your relationships") def edit_relationships(result): edit_options = tuple(result) selection = st.selectbox( "Select an entity relationship", edit_options ) st.write(f"This is a dialog {selection}") if st.button("Submit"): st.session_state.vote = 'pear' st.rerun() with tab1: st.write("This is the first tab") with tab2: query = st.text_input("Enter the entity", key='t2tinput') if st.button('send', key='t2button'): try: result = ['banana', 'apple', 'pear'] if st.button("Edit Relationships"): edit_relationships(result) except Exception as e: st.error(f"Error: {e}")  And I want that after the result list is received (from an API), that a button 'Edit relationships" appear so I can click on it and a modal appears. I tried this code but after clicking on 'Edit relationships', the modal doesn't appear. Please, could you point out what I am doing wrong?

Answer: It seems you need st.fragment:  When a user interacts with an input widget created inside a fragment, Streamlit only reruns the fragment instead of the full app. If run_every is set, Streamlit will also rerun the fragment at the specified interval while the session is active, even if the user is not interacting with your app.  Working with fragments:  Reruns are a central part of every Streamlit app.When users interact with widgets, your script reruns from top to bottom, and your app's frontend is updated. Streamlit provides several features to help you develop your app within this execution model. Streamlit version 1.37.0 introduced fragments to allow rerunning a portion of your code instead of your full script. As your app grows larger and more complex, these fragment reruns help your app be efficient and performant. Fragments give you finer, easy-to-understand control over your app's execution flow.  Full code: import streamlit as st st.title("Simulator") tab_names = ["tab1", "tab2"] tab1, tab2= st.tabs(tab_names) @st.dialog("Edit your relationships") def edit_relationships(result): edit_options = tuple(result) selection = st.selectbox( "Select an entity relationship", edit_options ) st.write(f"This is a dialog {selection}") if st.button("Submit"): st.session_state.vote = 'pear' st.rerun() @st.fragment def edit_btn(): if st.button("Edit Relationships"): edit_relationships(result) with tab1: st.write("This is the first tab") with tab2: query = st.text_input("Enter the entity", key='t2tinput') if st.button('send', key='t2button'): try: result = ['banana', 'apple', 'pear'] edit_btn() except Exception as e: st.error(f"Error: {e}")  Output:
Question: I want to check whether a library is installed and can be imported dynamically at runtime within an if statement to handle it properly. I've tried the following code: try: import foo print("Foo installed") except ImportError: print("Foo not installed")  It works as intended but does not seem like the most elegant method. I was thinking more of a method that returns a boolean indicating whether the library is installed, something like: Class.installed("foo") # returns a boolean 

Answer: Use importlib.util for a clean check. import importlib.util if importlib.util.find_spec("library_name") is not None: print("Installed")
Question: Let us assume we have this dataframe: df = pd.DataFrame.from_dict({1:{"a": 10, "b":20, "c":30}, 2:{"a":100, "b":200, "c":300}}, orient="index") Further, let us assume I want to apply a function to each row that adds 1 to the values in columns a and b def add(x): return x["a"] +1, x["b"] +1  Now, if I use the apply function to mod and overwrite the columns twice, some values are flipped: >>> df.loc[:, ["a", "b"]] = df[["a", "b"]].apply(lambda x: add(x), axis=1) >>> df a b c 1 11 101 30 2 21 201 300 >>> >>> df.loc[:, ["a", "b"]] = df[["a", "b"]].apply(lambda x: add(x), axis=1) >>> df a b c 1 12 22 30 2 102 202 300 >>>  Could somebody explain to me why b1 and a2 get flipped?

Answer: This is your original DataFrame:  a b c 1 10 20 30 2 100 200 300  Now, look at the output of df[['a', 'b']].apply(add, axis=1): df[['a', 'b']].apply(add, axis=1) 1 (11, 101) 2 (21, 201) dtype: object  This creates a Series of tuples, which means you have two items (11, 101) and (21, 201), and those are objects (tuples). The first item will be assigned to a, the second to b. Let see what happens if you were assigning two strings instead: df.loc[:, ['a', 'b']] = ['x', 'y'] a b c 1 x y 30 2 x y 300  The first item (x) gets assigned to a, the second (y) to b. Your unexpected behavior is due to a combination of two things:  you are ignoring the index with .loc[:, ...] the right hand side is a Series (of objects)  If you remove either condition, this wouldn't work: # let's assign on the columns directly df[['a', 'b']] = df[['a', 'b']].apply(add, axis=1) # KeyError: 0 # let's convert the output to list df[['a', 'b']] = df[['a', 'b']].apply(add, axis=1).tolist() # a b c # 1 11 21 30 # 2 101 201 300  In addition, your error only occurred because you had the same number of rows and columns in the selection. This would have raised an error with 3 columns: df.loc[:, ['a', 'b', 'c']] = df[['a', 'b', 'c']].apply(add, axis=1) # ValueError: Must have equal len keys and value when setting with an iterable  Take home message If you need to use a function with apply and axis=1 and you want to output several "columns", either convert the output to lists if you have the same columns as output: df[['a', 'b']] = df[['a', 'b']].apply(add, axis=1).tolist()  Or output a DataFrame by making the function return a Series: def add(x): return pd.Series({'a': x['a']+1, 'b': x['b']+1}) df[['a', 'b']] = df[['a', 'b']].apply(add, axis=1)  In any case, never use df.loc[:, ...] unless you know why you're doing this (i.e. you're purposely breaking the index alignment). Vectorial operations Of course, the above assumes you have complex, non-vectorized functions to use. If your goal is to perform a simple addition: # adding 1 to both a and b df[['a', 'b']] += 1 # adding 1 to a and 2 to b df[['a', 'b']] += [1, 2] # adding 1 to a and 2 to b, using add df[['a', 'b']] = df[['a', 'b']].add([1, 2]) # adding 1 to a and 2 to b, using a dictionary df[['a', 'b']] = df[['a', 'b']].add({'b': 2, 'a': 1})
Question: I am creating a parameterized Mock PyTest to test API behaviors. I am trying to simplify the test code by testing the instance modified behavior, e.g. throw and exception, and the default behavior, i.e. load JSON from file vs. calling REST API. I do not know how to add an array entry to represent the "default" mock behavior? @pytest.mark.parametrize( ("get_nearby_sensors_mock", "get_nearby_sensors_errors"), [ (AsyncMock(side_effect=Exception), {CONF_BASE: CONF_UNKNOWN}), (AsyncMock(side_effect=PurpleAirError), {CONF_BASE: CONF_UNKNOWN}), (AsyncMock(side_effect=InvalidApiKeyError), {CONF_BASE: CONF_INVALID_API_KEY}), (AsyncMock(return_value=[]), {CONF_BASE: CONF_NO_SENSORS_FOUND}), # What do I do here? # (AsyncMock(api.sensors, "async_get_nearby_sensors")) does not work as api is not in scope? # (AsyncMock(side_effect=None), {}) does not call the default fixture? (AsyncMock(), {}), ], ) async def test_validate_coordinates( hass: HomeAssistant, mock_aiopurpleair, api, get_nearby_sensors_mock, get_nearby_sensors_errors, ) -> None: """Test validate_coordinates errors.""" with ( patch.object(api, "async_check_api_key"), patch.object(api.sensors, "async_get_nearby_sensors", get_nearby_sensors_mock), ): result: ConfigValidation = await ConfigValidation.async_validate_coordinates( hass, TEST_API_KEY, TEST_LATITUDE, TEST_LONGITUDE, TEST_RADIUS ) assert result.errors == get_nearby_sensors_errors if result.errors == {}: assert result.data is not None else: assert result.data is None  How do I add a parameter for the "default behavior" of patch.object(api.sensors, "async_get_nearby_sensors") that will use the fixture to load data from canned JSON file? Why mock; async_validate_coordinates() calls async_check_api_key() that needs to be mocked to pass, and async_get_nearby_sensors() that is mocked with a fixture to return data from a JSON file. For ref this is the conftest.py file.

Answer: As a workaround, you could deal with any api-related case inside the function, where the api is known. For this single-use, I would use None to trigger the default case @pytest.mark.parametrize( ("get_nearby_sensors_mock", "get_nearby_sensors_errors"), [ # [...] (None, {}), ], ) async def test_validate_coordinates( hass: HomeAssistant, mock_aiopurpleair, api, get_nearby_sensors_mock, get_nearby_sensors_errors, ) -> None: """Test validate_coordinates errors.""" if get_nearby_sensors_mock is None: get_nearby_sensors_mock = AsyncMock(api.sensors, "async_get_nearby_sensors") with (patch.object( # [...]
Question: I want to vertically merge two polars.LazyFrames in order to avoid collecting both LazyFrames beforehand, which is computationally expensive. I have tried extend(), concat(), and vstack() but none of them are implemented for LazyFrames. Maybe I am missing the point about LazyFrames by trying to perform this operation, but I am aware that join() works, which would also alter the dataframe's structure.

Answer: pl.concat can be used with LazyFrames: >>> lf = pl.LazyFrame({"x": [1, 2]}) >>> pl.concat([lf, lf]).collect() shape: (4, 1) ┌─────┐ │ x │ │ --- │ │ i64 │ ╞═════╡ │ 1 │ │ 2 │ │ 1 │ │ 2 │ └─────┘