eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm trying to create an Altair faceted bar chart with lines that are represents other measure and would better use of a second y-axis on the right side. I don't know if it is possible using Atair. The code is as follows: import altair as alt import pandas as pd data = { 'Date': ['2023-01-01', '2023-01-02', '2023-01-01', '2023-01-02', '2023-01-01', '2023-01-02', '2023-01-01', '2023-01-02', '2023-01-01', '2023-01-02', '2023-01-01', '2023-01-02', '2023-01-01', '2023-01-02', '2023-01-01', '2023-01-02'], 'Value': [50, 200, 300, 150, 250, 350, 200, 200, 10, 20, 15, 20, 20, 30, 20, 30], 'DESCR': ['A', 'A', 'A', 'A', 'B', 'B', 'B', 'B', 'C', 'C', 'C', 'C', 'D', 'D', 'D', 'D'], 'Company': ['X', 'X', 'Y', 'Y', 'X', 'X', 'Y', 'Y', 'X', 'X', 'Y', 'Y', 'X', 'X', 'Y', 'Y'], } source = pd.DataFrame(data) bars = alt.Chart(source).mark_bar().transform_filter(f"(datum.DESCR=='A') | (datum.DESCR=='B')").encode( x='Date:N', y='Value:Q', color=alt.Color('DESCR:N', legend=alt.Legend(title=None)) ) lines = alt.Chart(source).mark_line().transform_filter(f"(datum.DESCR=='C') | (datum.DESCR=='D')").encode( x='Date:N', y='Value', stroke=alt.Stroke('DESCR', legend=alt.Legend(title=None), scale=alt.Scale(scheme='redblue')) ) chart = (bars+lines).facet(column='Company') chart  I have tried .resolve_scale(y='independent') after .facet, but it doesn't show the second axis on the right side. Any help would be appreciated.

Answer: As per https://github.com/vega/vega-lite/issues/4373#issuecomment-617153232, you need some additional resolves: chart = ( (bars+lines) .resolve_scale(y='independent') # Create dual axis .facet(column='Company') .resolve_axis(y='independent') # Make sure dual axis works with facet (redraws the axis for each subplot) .resolve_scale(y='shared') # Set the y-max across facets to be the same (the default when not having dual axes) )
Question: >>> import selenium >>> selenium.webdriver Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'selenium' has no attribute 'webdriver' >>> from selenium import * >>> webdriver Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'webdriver' is not defined >>> from selenium import webdriver >>> webdriver <module 'selenium.webdriver' from 'C:\\Python312\\Lib\\site-packages\\selenium\\webdriver\\__init__.py'>  In this code why selenium doesn't import webdriver unless if asked explicitly. According to my current understanding, if not asked, in python all classes/functions should load. then why webdriverd was not imported when we import all classes/functions from module unless asked explicitly. I was expecting webdriver to be load when i load selenium module or all classes/functions from selenium module but webdriver wasn't imported and is only loaded when asked explicitly. Edit: To add more context, I am using Windows OS. Also, I don't have a file named selenium.py in the current directory, which could conflict with the selenium module.

Answer:  According to my current understanding, if not asked, in python all classes/functions should load.  The flaw in your understanding is that selenium.webdriver is not a class or function. In reality, selenium.webdriver is a module, and it has to be imported, rather than used as if it was an object. Note that importing the selenium module doesn't import its child modules; i.e. webdriver, webdriver.common and so on. If you thought that it would or should do that, your understanding is incorrect. Try getting started with Selenium by copying the "simple usage" example provided in https://selenium-python.readthedocs.io/getting-started.html#simple-usage: from selenium import webdriver from selenium.webdriver.common.keys import Keys from selenium.webdriver.common.by import By driver = webdriver.Firefox() driver.get("http://www.python.org") assert "Python" in driver.title elem = driver.find_element(By.NAME, "q") elem.clear() elem.send_keys("pycon") elem.send_keys(Keys.RETURN) assert "No results found." not in driver.page_source driver.close()  It seems to me that you need to do some more work on your understanding of Python modules, how they work, and how you use them.
Question: error info:  .\tools.c(17775): error C2069: Cast from "void" to non-void .\tools.c(17775): error C2036: "void *" : unknown size  my code: This function attempts to free the memory of one-dimensional array or two-dimensional array, but do not work. cdef void free_memory(void* arr, int ndim, int rows): cdef int i if ndim == 1: free(<double*>arr) elif ndim == 2: for i in range(rows): free(<double*>arr[i]) free(<double**>arr) else: raise ValueError("Unsupported number of dimensions (ndim must be 1 or 2).")  example: cdef double** array = <double**> malloc(n * sizeof(double*)) free_memory(array , 2, n)  I have tried to modify it many times, including using gpt assistance, but it still doesn't work.

Answer: This: free(<double*>arr[i])  is attempting to index a void*, and then cast the result to the type you're expecting. But you can't index a void*. The compiler doesn't have any information about what type arr points to. You're casting too late. What you need to do is cast arr itself to the right type, and then index that: free((<double**>arr)[i])   Aside from that, if you want to free the row pointers, you need to actually have row pointers. Your example: cdef double** array = <double**> malloc(n * sizeof(double*)) free_memory(array , 2, n)  doesn't actually allocate any memory for the rows.
Question: I have a generic class with a function that returns a Pydantic model where one of the fields is the generic type. What follows is a code snippet that defines two classes the generic GetValue and GetInt. I don't understand why the behavior of GetValue[int] is not the same as GetInt. from typing import Any, Generic, TypeVar from pydantic import BaseModel X = TypeVar("X") class TestModel(BaseModel, Generic[X]): a: X T = TypeVar("T") class GetValue(Generic[T]): @classmethod def get_value(cls, x: Any) -> TestModel[T]: return TestModel[T](a=x) class GetInt: @classmethod def get_value(cls, x: Any) -> TestModel[int]: return TestModel[int](a=x) res = GetValue[int].get_value("1") assert type(res.a) == str res_2 = GetInt.get_value("1") assert type(res_2.a) == int assert res.a != res_2.a # I want these two to be equal, not sure why they are different. x: Any = "1" res_3 = TestModel[int](a=x) assert type(res_3.a) == int  Inspecting things in my debugger I see that in this function call GetValue[int].get_value("1") T is just a TypeVar. This causes Pydantic to fall back on an Any annotation for a so it stays as a str. res_3 shows that when the type is supplied to the Pydantic class it will convert the string to an int. I'm not sure why this construction isn't working how I'd expect. Still new to python's typing system, was hoping someone could shed light on this. Python version: Python 3.12.5 pydantic version: 2.10.0

Answer: Well, Python doesn't really support a simple way to determine the type of a TypeVar at runtime. pydantic models are a bit special because they override the standard behaviour of generic classes such that they can more or less easily determine a generic type. However, I just realized that in your code you're not trying around with a pydantic generic model (which I thought, when I wrote the comment yesterday). Instead, you want that for any "standard" generic class, right? You may want to inform yourself about all this stuff with __args__ and _GenericAlias in the typing module. But I actually created a package (python-generics) some time ago to solve this problem. With that, the solution would look like: from typing import Any, Generic, TypeVar from generics import get_filled_type from pydantic import BaseModel X = TypeVar("X") class TestModel(BaseModel, Generic[X]): a: X T = TypeVar("T") class GetValue(Generic[T]): @classmethod def get_value(cls, x: Any) -> TestModel[T]: return TestModel[get_filled_type(cls, GetValue, T)](a=x)  One last note: It is generally possible for type checkers like mypy to narrow the type for generic classes through e.g. constructors elements. Meaning that the generic type doesn't necessarily have to be explicitly defined in the code to pass the type check. But if you want the solution to work you must always define the type explicitly.
Question: I have some function that accepts a dict-like object. from typing import Dict def handle(x: Dict[str, str]): pass # Do some processing here...  I still get a type warning if I try passing a Shelf to the function, even though the function supports it. How to I specify the type of a dict-like object?

Answer: Use collections.abc.Mapping (typing.Mapping is deprecated): from collections.abc import Mapping def handle(x: Mapping[str, str]): pass # Do some processing here...
Question: I want to plot the slope field for: 0.5*sin(0.5*pi*x)*sqrt(y+7) import numpy as np import matplotlib.pyplot as plt # Specify the grid of dots x = np.arange(-3,3,0.3) y = np.arange(-2,4,0.3) X, Y = np.meshgrid(x,y) # Create unit vectors at each dot with correct slope dy = 0.5*(np.sin(x*np.pi*0.5))*np.sqrt(y+7) dx = np.ones(dy.shape) norm = np.sqrt(X**2 + Y**2) dyu = dy/norm dxu = dx/norm # Plot everything plt.quiver(X,Y,dxu,dyu,) plt.show()  and I got the second image below. I am trying to replicate the first image. How can I make uniform slope lines like that? Why am I getting variable-length lines anyway?  

Answer: So, the first thing to do is remove the arrowheads, which @jasonharper shows in the comments can be done by adding these options to your quiver call: headwidth=0, headlength=0, headaxislength=0. Next is to deal with the length. You're currently normalizing by X and Y when you should be normalizing by dx and dy. I would actually redefine dx and dy to use X and Y so they have the same shapes. And you will also want to make the pivot="mid" change that @jasonharper mentioned in their comment. With these changes, your code would look like this: import numpy as np import matplotlib.pyplot as plt plt.close("all") x = np.arange(-3, 3, 0.3) y = np.arange(-2, 4, 0.3) X, Y = np.meshgrid(x, y) dX = np.ones_like(Y) dY = 0.5*np.sin(X*np.pi*0.5)*np.sqrt(Y+7) norm = np.sqrt(dX**2 + dY**2) dXu = dX/norm dYu = dY/norm plt.quiver(X, Y, dXu, dYu, headwidth=0, headlength=0, headaxislength=0, pivot="mid") plt.show()  Result:   To control the length, you'll need to change two things. The first thing would be to multiply dX and dY by the desired length. The second thing would be to add the following options to quiver: angles="xy", scale_units="xy", scale=1 to make sure the vectors are the desired length. import numpy as np import matplotlib.pyplot as plt plt.close("all") x = np.arange(-3, 3, 0.3) y = np.arange(-2, 4, 0.3) X, Y = np.meshgrid(x, y) dX = np.ones_like(Y) dY = 0.5*np.sin(X*np.pi*0.5)*np.sqrt(Y+7) norm = np.sqrt(dX**2 + dY**2) dXu = dX/norm dYu = dY/norm L= 0.2 dXs = L*dXu dYs = L*dYu plt.quiver(X, Y, dXs, dYs, headwidth=0, headlength=0, headaxislength=0, angles="xy", scale_units="xy", scale=1, pivot="mid") plt.show()
Question: Can somebody tell me what am I doing wrong? The bot sends me an inline keyboard, but after clicking the button, I do not receive any callbacks (even the logger doesn't send anything). Also, if I pass URL to inlineKeyboardButton it works properly @bot.message_handler() def add_city(message): if user_status[message.chat.id] == Statuses.AddCity: logger.info("User adding the city") request = { "name": message.text } response = requests.post(f'{api_url}/api/users/{message.chat.id}/add-city', json=request, headers=headers) if response.status_code == 200: logger.info("User got city. Waiting for confirmation") data = response.json() user_status[message.chat.id] = Statuses.ConfirmCity markup = InlineKeyboardMarkup() markup.add(InlineKeyboardButton("Confirm", callback_data="Confirm")) markup.add(InlineKeyboardButton("Deny", callback_data="Deny")) bot.send_message(message.chat.id, f'{data.get("name")}') bot.send_message(message.chat.id, f'Підтверди що це саме те місто, яке ти хотів додати:', reply_markup=markup) else: logger.info("Bad Request") bot.send_message(message.chat.id, f'Виникла помилка при додаванні міста :(') @bot.callback_query_handler(func=lambda call: call.data in ("Confirm", "Deny")) def handle_city_confirmation(call): logger.info(call) if user_status[call.message.chat.id] == Statuses.ConfirmCity: if call.data.split(' ')[0] == "Confirm": bot.send_message(call.message.chat.id, 'success') else: bot.send_message(call.message.chat.id, "deny") else: bot.send_message(call.message.chat.id, "fail") 

Answer: The solution was updating the token. Hope it will help someone
Question: What is the faster numpythonic way of this normalization: def normalize_vector(x, b, axis): """ Normalize real vector x and outputs an integer vector y. Parameters: x (numpy.ndarray): Input real vector. (batch_size, seq_len) b (int): Unsigned integer defining the scaling factor. axis (int/None): if None, perform flatenned version, if axis=-1, perform relative normalization across batch. Returns: numpy.ndarray: Integer vector y. """ # Find the maximum absolute value in x m = np.max(np.abs(x)) # Process each element in x y = [] for xi in x: if xi > 0: y.append(int((2**b - 1) * xi / m)) elif xi < 0: y.append(int(2**b * xi / m)) else: y.append(0) return np.array(y)  Can np.digitize make it faster? I have similar question, but it's not about NumPy. I'm also expecting it supports axis parameter for batch vector.

Answer: there is np.piecewise to transform data based on multiple conditions. def normalize_vector2(x, b, axis): # Step 1: Find the maximum absolute value in `x` m = np.max(np.abs(x), axis=axis) y = np.piecewise(x, [x > 0, x < 0], [ lambda xi: ((2**b - 1) * xi / m), lambda xi: (2**b * xi / m) ]) return y.astype(int)  if your paths are close then you can just simplify it with multiplies. def normalize_vector3(x, b, axis): # Step 1: Find the maximum absolute value in x m = np.max(np.abs(x), axis=axis, keepdims=True) m[m==0] = 1 y = (2**b - 1 * (x > 0)) * x / m return y.astype(int)  comparison: import numpy as np import time def normalize_vector2(x, b): # Step 1: Find the maximum absolute value in `x` m = np.max(np.abs(x)) y = np.piecewise(x, [x > 0, x < 0], [ lambda xi: ((2**b - 1) * xi / m), lambda xi: (2**b * xi / m) ]) return y.astype(int) def normalize_vector3(x, b, axis): # Step 1: Find the maximum absolute value in x m = np.max(np.abs(x), axis=axis, keepdims=True) m[m==0] = 1 y = (2**b - 1 * (x > 0)) * x / m return y.astype(int) def normalize_vector(x, b): # Find the maximum absolute value in x m = np.max(np.abs(x)) # Process each element in x y = [] for xi in x: if xi > 0: y.append(int((2**b - 1) * xi / m)) elif xi < 0: y.append(int(-2**b * xi / m)) else: y.append(0) return np.array(y) for elements in [10, 100, 1000, 10000]: iterations = int(100000 / elements) x = np.random.random(elements) * 256-128 t1 = time.time() for i in range(iterations): normalize_vector(x,7) t2 = time.time() for i in range(iterations): normalize_vector2(x, 7) t3 = time.time() for i in range(iterations): normalize_vector3(x, 7, 0) t4 = time.time() print(f"{(t2-t1)/iterations:.7f}, {elements} elements python") print(f"{(t3-t2)/iterations:.7f}, {elements} elements numpy") print(f"{(t4-t3)/iterations:.7f}, {elements} elements numpy maths")  0.0000109, 10 elements python 0.0000331, 10 elements numpy 0.0000158, 10 elements numpy maths 0.0000589, 100 elements python 0.0000399, 100 elements numpy 0.0000168, 100 elements numpy maths 0.0005812, 1000 elements python 0.0000515, 1000 elements numpy 0.0000255, 1000 elements numpy maths 0.0045110, 10000 elements python 0.0003255, 10000 elements numpy 0.0001083, 10000 elements numpy maths  numpy is slower than pure python for small lists (mostly < 50 elements).
Question: I have a PySpark dataframe like this but with a lot more data:    user_id event_date     123 '2024-01-01 14:45:12.00'   123 '2024-01-02 14:45:12.00'   456 '2024-01-01 14:45:12.00'   456 '2024-03-01 14:45:12.00'    I drop duplicates of users, leaving the last event. I am using something like this: df = df.orderBy(['user_id', 'event_date'], ascending=False).dropDuplicates(['user_id'])  When I was searching for the solution to some other problem, I found information that this approach may be non-deterministic. Am I doing it wrong? Should I use window functions instead?

Answer: When you call dropDuplicates() without passing any columns to it - it just drops the identical rows, no matter what, for all columns (so we may kind of call it "deterministic" - as all columns will have same values in different rows being dropped and only one is kept - doesn't matter which one). This non-deterministic behaviour of dropDuplicates(), when passing a subset of columns, is a known issue and using orderBy() doesn't enforce determinism either because it's not guaranteed that the ordering will be maintained after applying dropDuplicates() due to Spark's internal implementations (partitioning, logical and physical plans etc). It's recommended to use Windows functions instead: from pyspark.sql import functions as F from pyspark.sql.window import Window window_spec = Window.partitionBy("user_id").orderBy(F.col("event_date").desc()) df_with_rank = df.withColumn("row_number", F.row_number().over(window_spec)) df_last_event = df_with_rank.filter(F.col("row_number") == 1).drop("row_number")
Question: How does Python ThreadPoolExecutor switch between concurrent threads? In the case of the async/awaint event-loop, the switching between different pieces of the code happens at the await calls. Does the ThreadPoolExecutor run each submitted task for a random amount of time> Or until something somewhere calls Thread.sleep()? Or the OS temporarily switches to do something else and the thread is forced to release the GIL thus allowing some other thread to grab the GIL next time?

Answer:  Does the ThreadPoolExecutor run each submitted task for a random amount of time?  No. The executor runs a thread pool with a work queue. When you add a new task, a thread will pick it up and run it to completion. Individual threads do not switch tasks before the previous task has completed. As long as the global interpreter lock (GIL) is still around, only one thread can execute Python bytecode at any given moment (within one process; not counting multiple sub-interpreters). Since Python-3.2 the GIL switches threads every 5 ms (configurable) as opposed to every 100 instructions like it was before. For comparison, the Linux kernel typically runs its scheduler with time slices between 0.75 and 6 ms.  Or until something somewhere calls Thread.sleep()?  Every system call that may block / sleep and every call into compiled code that may run for extended periods of time is typically written to release the GIL. This works in addition to the 5 ms time slice and is the intended work regime for thread pools. If the background threads spend most of their time with the GIL released waiting on IO or expensive library calls (such as numpy), they parallelize effectively. If they mostly execute Python bytecode, they just compete with the main event loop for the GIL. That can still work fine if the event loop is mostly idle but it isn't exactly best-practice and it doesn't scale well.  Or the OS temporarily switches to do something else and the thread is forced to release the GIL thus allowing some other thread to grab the GIL next time?  The OS doesn't know the GIL. If the OS interrupts the thread currently holding the GIL, it will continue to hold it. No other thread can acquire it during that time. Once the thread wakes back up it will likely have exceeded its 5 ms time slice and will release the GIL at the next opportunity.
Question: Assuming the following models: class Store(models.Model): name = models.CharField(max_length=255) class Stock(models.Model): name = models.CharField(max_length=255) store = models.ForeignKey(Store, on_delete=models.PROTECT) class Consignment(models.Model): cost = models.FloatField() class Order(models.Model): quantity = models.IntegerField() stock = models.ForeignKey(Stock, on_delete=models.PROTECT) consignment = models.ForeignKey(Consignment, on_delete=models.PROTECT)  How to create a queryset of all 'consignments' of all instances of a specific 'stores' queryset like so: target_stores = Store.objects.filter(name__startswith="One") consignments = target_stores.consignments.all() 

Answer: You query in reverse by spanning over multiple relations with the __ separator: Consignment.objects.filter(order__stock__store__name__startswith='One')
Question: An enum is declared in an imported package and identically in the importer. Same value, but Python treats the imported enum value as different for some reason. Package 1 is a parser that I wrote which outputs a dictionary containing some values from this enum declared in the parser package: class NodeFace(Enum): TOP = 0 BOTTOM = 1 RIGHT = 2 LEFT = 3  So, it parses along and then maps some text to these values in a dictionary that will be accessible to the importer. In case it matters, the parser is built using Python 3.13. Now the importer, built using Python 3.12, also declares the same enum in its own local files. Identical to above. Additionally, I use this dictionary to find opposite sides, declared in the importer: OppositeFace = { NodeFace.TOP: NodeFace.BOTTOM, NodeFace.BOTTOM: NodeFace.TOP, NodeFace.LEFT: NodeFace.RIGHT, NodeFace.RIGHT: NodeFace.LEFT }  In the importer I assign enum values from the parser output dict to variables tface and pface. And, I test to make sure they look right by printing to console and sure enough I get values like: <NodeFace.BOTTOM: 1>. In fact, I'm looking at them in the debugger, no problem so far. Now the fun starts... if OppositeFace[tface] == pface:  This fails on a KeyError on the OppositeFace lookup. BUT, if I do the lookup in the debugger console with: OppositeFace[NodeFace.TOP], it works just fine. I did all kinds of tests and all I could figure is that the problem was in defining the enum twice, once in the parser and again in the importer and, despite all the values matching in the debugger and console, some internal value is different and causing the dict lookup to fail. My solution was to eliminate the Enum from my parser and just pass strings 'TOP', 'BOTTOM', etc. Then, on the importer side I do OppositeFace[NodeFace[tface]] where tface is now the string and it works fine. Can anyone tell me why exactly this happens? Just curious at this point.

Answer: This is just how Python normally works. Defining two classes the same way doesn't make them the same class, or make their instances equal. Even if they print the same, Python doesn't compare objects based on how they print. You have two separate enums with separate members. Unless you implement different behavior (which you didn't), enums inherit the default identity-based __eq__ and __hash__ implementations from object. That means an enum member is only equal to itself. The only relevant enum magic is that writing TOP = 0 in an enum definition makes TOP an instance of the enum class, rather than the int 0. Integers compare by numeric value rather than identity (and also there's a CPython implementation detail that would make the ints the same object anyway), so if your enum members really were just ints, they would compare equal. They're not, though.