eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: When loading securities quote data, the price values ​​look like this: Where units is the integer part, nano is the fractional part.  | close | volume | time | {'units': 270, 'nano': 190000000} | 51642 | 2023-12-29 07:30:00+00:00 | {'units': 270, 'nano': 930000000} | 47910 | 2023-12-29 08:00:00+00:00 | {'units': 271, 'nano': 0} | 49376 | 2023-12-29 08:30:00+00:00  How can I fix the "close" column so that the data looks like this?  | close | volume | time | 270.19 | 51642 | 2023-12-29 07:30:00+00:00 | 270.93 | 47910 | 2023-12-29 08:00:00+00:00 | 271 | 49376 | 2023-12-29 08:30:00+00:00  I tried to sum the values as follows, but nothing came out. def cost_money(v): return v.units + v.nano / 1e9 

Answer: Convert dictionaries to DataFrame by json_normalize and then create new column: v = pd.json_normalize(df['close']) df['close'] = v.units + v.nano / 1e9  But if there are strings repr of dictionaries first convert them: import ast v = pd.json_normalize(df['close'].apply(ast.literal_eval)) df['close'] = v.units + v.nano / 1e9   print (df) close volume time 0 270.19 51642 2023-12-29 07:30:00+00:00 1 270.93 47910 2023-12-29 08:00:00+00:00 2 271.00 49376 2023-12-29 08:30:00+00:00  Youe solution: import ast def cost_money(v): #if necessary convert string repr to dicts v = ast.literal_eval(v) return v['units'] + v['nano'] / 1e9 df['close'] = df['close'].apply(cost_money)
Question: I have a dataset where I would like for some values to become column headers, as well as a crosstab format. data year qtr ID type growth re nondd_re se_re or 2024 2024Q1 NY aa 3.18 1.14 0 0 0 2024 2024Q2 NY aa 2.1 1.14 0 0 0 2024 2024Q1 NY dd 6.26 3.07 3.07 0 0 2024 2024Q2 NY dd 4.13 3.07 3.07 0 0 2024 2024Q1 CA aa 0 0 0 0 0 2024 2024Q2 CA aa 0.03 0 0 0 0 2024 2024Q1 CA dd 0 0 0 0 0 2024 2024Q2 CA dd 0.06 0 0 0 0  desired ID type type 2024Q1 2024Q2 NY growth dd 6.26 4.13 NY nond_ re dd 3.07 3.07 NY se_re dd 0 0 NY or dd 0 0 NY re dd 3.07 3.07 NY growth aa 3.18 2.1 NY nond_ re aa 0 0 NY se_re aa 0 0 NY or aa 0 0 NY re aa 1.14 1.14 CA growth dd 0 0.6 CA nond_ re dd 0 0 CA se_re dd 0 0 CA or dd 0 0 CA re dd 0 0 CA growth aa 0 0.3 CA nond_ re aa 0 0 CA se_re aa 0 0 CA or aa 0 0 CA re aa 0 0  doing # Melt the dataframe to transform metrics columns into rows melted_df = df.melt(id_vars=["year", "qtr", "ID", "type"], var_name="type", value_name="value") # Pivot the melted dataframe pivot_df = melted_df.pivot_table(index=["ID","type"], columns="qtr", values="value", fill_value=0) # Reset index to turn multi-index into columns pivot_df = pivot_df.reset_index()  The issue is that all of the values aren't getting. The above code produces output with missing values Any suggestion is appreciated

Answer: You can try to set_index() + stack()/unstack(): out = ( df.set_index(["year", "qtr", "ID", "type"]) .stack() .unstack("qtr") .reset_index() .rename(columns={"level_3": "type2"}) .rename_axis(columns=None, index=None) ) print(out)  Prints:  year ID type type2 2024Q1 2024Q2 0 2024 CA aa growth 0.00 0.03 1 2024 CA aa re 0.00 0.00 2 2024 CA aa nondd_re 0.00 0.00 3 2024 CA aa se_re 0.00 0.00 4 2024 CA aa or 0.00 0.00 5 2024 CA dd growth 0.00 0.06 6 2024 CA dd re 0.00 0.00 7 2024 CA dd nondd_re 0.00 0.00 8 2024 CA dd se_re 0.00 0.00 9 2024 CA dd or 0.00 0.00 10 2024 NY aa growth 3.18 2.10 11 2024 NY aa re 1.14 1.14 12 2024 NY aa nondd_re 0.00 0.00 13 2024 NY aa se_re 0.00 0.00 14 2024 NY aa or 0.00 0.00 15 2024 NY dd growth 6.26 4.13 16 2024 NY dd re 3.07 3.07 17 2024 NY dd nondd_re 3.07 3.07 18 2024 NY dd se_re 0.00 0.00 19 2024 NY dd or 0.00 0.00
Question: I know what a single asterisk (*) means in a function definition. See, for example, this question. However, the documentation of warnings.warn contains the entry \*. What does this symbol combination mean in a function definition?

Answer: A typo, up to 3.11 it's not there warnings.warn(message, category=None, stacklevel=1, source=None): warnings.warn(message, category=None, stacklevel=1, source=None) Issue a warning, or maybe ignore it or raise an exception. The category argument, if given, must be a warning category class; it defaults to UserWarning. Alternatively, message can be a Warning instance, in which case category will be ignored and message.__class__ will be used. In this case, the message text will be str(message). This function raises an exception if the particular warning issued is changed into an error by the warnings filter. The stacklevel argument can be used by wrapper functions written in Python, like this: def deprecation(message): warnings.warn(message, DeprecationWarning, stacklevel=2) This makes the warning refer to deprecation()’s caller, rather than to the source of deprecation() itself (since the latter would defeat the purpose of the warning message). source, if supplied, is the destroyed object which emitted a ResourceWarning. Changed in version 3.6: Added source parameter.  Try it: def func(\*): pass  Result: ERROR! File "<string>", line 1 def func(\*): ^ SyntaxError: unexpected character after line continuation character  and : def func(po = 4): print('po = ' , po) pass a = func( 3) a = func(po = 3) def func(*, po = 4): print('po = ' , po) pass a = func(po = 3) a = func(3)  output : ERROR! po = 3 po = 3 po = 3 Traceback (most recent call last): File "<string>", line 18, in <module> TypeError: func() takes 0 positional arguments but 1 was given >  See What does a bare asterisk do in a parameter list? What are "keyword-only" parameters? for explanation :  Bare * is used to force the caller to use named arguments  and PEP 570 :
Question: I would like to write a statically typed Python function that accepts a general type and a string and that returns a value of the given type, derived from the given string. For example: >>> parse(int | None, "5") 5 >>> parse(int | None, "") is None True  Here's a Python 3.12 attempt (which has a problem in the type signature since union types can't be assigned to variables of type type): def parse[a](x: type[a], v: str) -> a: if x in (bool, bool | None): return bool(v) if x in (float, float | None): return float(v) if x in (int, int | None): return int(v) if x in (str, str | None): return v raise ValueError  However, this code yields errors with Pyright 1.1.349: error: Expression of type "bool" cannot be assigned to return type "a@parse" error: Expression of type "float" cannot be assigned to return type "a@parse" error: Expression of type "int" cannot be assigned to return type "a@parse"  How can this be fixed?

Answer: In short, this is a problem of parametricity. Your function is not the same for every possible type A. With the signature def parse[A](x: type[A], v: str) -> A: ...  the type checker has no idea what A is, only that a value of type A is required. As such, the only way to get such a value is to call x. The following trivial definition would type-check: def parse[A](x: type[A], v: str) -> A: return x()  Your function, however, tries to return a value of type bool, or type str, or type int, etc, which (statically speaking) is distinct from a value of type A, even if A is bound at runtime to bool or str or int. Type narrowing does not apply to a generic type. One workaround is to pass not the type itself, but a string representing the type, and using typing.overload to implement ad-hoc polymorphism rather than parametric polymorphism. from typing import overload, Literal @overload def parse(x: Literal["str"], v: str) -> str: ... @overload def parse(x: Literal["float"], v: str) -> float: ... @overload def parse(x: Literal["int"], v: str) -> int: ... @overload def parse(x: Literal["bool"], v: str) -> bool: ... def parse(x, v): match x: case "str": return v case "int": return int(v) case "float": return float(v) case "bool": return v.lower() == "true" case _: raise ValueError reveal_type(parse("int", "3")) reveal_type(parse("float", "3.0")) reveal_type(parse("bool", "True")) reveal_type(parse("str", "foo")) # The following is a type error, but you were going to # raise a runtime error anyway. reveal_type(parse("complex", "3.9"))
Question: I am transfering code between Matlab and Python. However when I use scipy.special.gammainc I get different values than when I use gammainc in Matlab. My Python code is: from scipy.special import gammainc gammainc([1, 2, 3, 4], 5) array([0.99326205, 0.95957232, 0.87534798, 0.73497408])  However in Matlab I have: gammainc([1,2,3,4],5,'lower') [0.00365984682734371, 0.0526530173437111, 0.184736755476228, 0.371163064820127]  I am trying to change my Python version to give me the same version as the Matlab code. I have tried instead: from scipy.special import gammaincc gammaincc([1, 2, 3, 4], 5) array([0.00673795, 0.04042768, 0.12465202, 0.26502592])  This is closer but it is still not what I am looking for.

Answer: There are two issues with your code:  Matlab's gammainc function expects the input arguments in reverse order compared to the usual mathematical notation. So it expects the integral limit first, then the exponent. Scipy's gammaincc is the upper incomplete Gamma function, not the lower. For the lower version you need to use gammainc.  So, to reproduce Matlab's result in Python with Scipy, use from scipy.special import gammainc print(gammainc(5, [1, 2, 3, 4]))  This gives [0.00365985 0.05265302 0.18473676 0.37116306]
Question: I have a pandas dataframe and the opposite question to Drop all rows before first occurrence of a value Instead of drop all rows before the first occurence of a value, I want to keep the rows before the first occurence of a value. so for example, i want to keep all the rows before the occurence 1 in the column count: Year ID Count 1997 1 0 1998 2 0 1999 3 1 2000 4 0 2001 5 1  leads to: Year ID Count 1997 1 0 1998 2 0 

Answer: Use cummax and negate its output (~) to form a boolean Series for boolean indexing: out = df[~df['Count'].eq(1).cummax()]  Output:  Year ID Count 0 1997 1 0 1 1998 2 0  Intermediates:  Year ID Count eq(1) cummax ~ 0 1997 1 0 False False True 1 1998 2 0 False False True 2 1999 3 1 True True False 3 2000 4 0 False True False 4 2001 5 1 True True False  Variant, if you are sure that there is at least one 1: out = df.iloc[:np.argmax(df['Count'].eq(1))]
Question: I have two list of dictionaries in the format: list1 = [ {"time": "2024-01-29T18:32:24.000Z", "value": "abc"}, {"time": "2024-01-30T19:47:48.000Z", "value": "def"}, {"time": "2024-01-30T19:24:20.000Z", "value": "ghi"} ] list2 = [ {"time": "2024-01-30T18:34:44.000Z", "value": "xyz"}, {"time": "2024-01-30T19:47:48.000Z", "value": "pqr"}, {"time": "2024-01-30T19:24:20.000Z", "value": "jkl"} ]  Requirement : I need to compare value of "time" key of each dictionary in list1 with "time" key of each dictionary in list2 and if they are equal, will need to form a key value pair dictionary for subsequent lookups. In the above, list1 and list2 have 2 dictionaries with same time : "2024-01-30T19:47:48.000Z"  "2024-01-30T19:24:20.000Z"  I need to combine value of these 2 to form output as below : Output : {"def" : "pqr", "ghi" : "jkl"}  using itertools.product function , i am able to list as below so far but not in the key value pair dictionary as required . import itertools list1 = [{"time": "2024-01-29T18:32:24.000Z", "value": "abc"}, {"time": "2024-01-30T19:47:48.000Z", "value": "def"}, {"time": "2024-01-30T19:24:20.000Z", "value": "ghi"}] list2 = [{"time": "2024-01-30T18:34:44.000Z", "value": "xyz"}, {"time": "2024-01-30T19:47:48.000Z", "value": "pqr"}, {"time": "2024-01-30T19:24:20.000Z", "value": "jkl"}] output = [f"{x['value']} : {y['value']}" if x['time'] == y['time'] else None for (x, y) in itertools.product(list1, list2)] print(output)  Current Output : [None, None, None, None, 'def : pqr', None, None, None, 'ghi : jkl']  I'm wondering if i can use lambda function to achieve output as : {"def" : "pqr", "ghi" : "jkl"} Any help or suggestions is appreciated.

Answer: dict comprehension has an if clause (at the end), to also filter out elements, like this: output = {x['value']: y['value'] for x, y in itertools.product(list1, list2) if x['time'] == y['time']}  Output: {'def': 'pqr', 'ghi': 'jkl'}
Question: I'm trying to vectorize the following "power-of-grad" function so that it accepts multiple orders: (see here) def grad_pow(f, order, argnum): for i in jnp.arange(order): f = grad(f, argnums=argnum) return f  This function produces the following error after applying vmap on the argument order: jax.errors.ConcretizationTypeError: Abstract tracer value encountered where concrete value is expected: traced array with shape int32[]. It arose in the jnp.arange argument 'stop'  I have tried writing a static version of grad_pow using jax.lax.cond and jax.lax.scan, following the logic here: def static_grad_pow(f, order, argnum): order_max = 3 ## maximum order def grad_pow(f, i): return cond(i <= order, grad(f, argnum), f), None return scan(grad_pow, f, jnp.arange(order_max+1))[0] if __name__ == "__main__": test_func = lambda x: jnp.exp(-2*x) test_func_grad_pow = static_grad_pow(jax.tree_util.Partial(test_func), 1, 0) print(test_func_grad_pow(1.))  Nevertheless, this solution still produces an error:  return cond(i <= order, grad(f, argnum), f), None TypeError: differentiating with respect to argnums=0 requires at least 1 positional arguments to be passed by the caller, but got only 0 positional arguments.  Just wondering how this issue can be resolved?

Answer: The fundamental issue with your question is that a vmapped function cannot return a function, it can only return arrays. All other details aside, that precludes any possibility of writing a valid function that does what you intend. There are alternatives: for example, rather than attempting to create a function that will return a function, you could instead create a function that accepts arguments and applies that function to those arguments. In that case, you'll run into another issue: if n is traced, there is no way to apply grad n times. JAX transformations like grad are evaluated at trace-time, and traced values like n are not available until runtime. One way to work around this is to pre-define all the functions you're interested in, and to use lax.switch to choose between them at runtime. The result would look something like this: import jax import jax.numpy as jnp from functools import partial @partial(jax.jit, static_argnums=[0], static_argnames=['argnum', 'max_order']) def apply_multi_grad(f, order, *args, argnum=0, max_order=10): funcs = [f] for i in range(max_order): funcs.append(jax.grad(funcs[-1], argnum)) return jax.lax.switch(order, funcs, *args) order = jnp.arange(3) x = jnp.ones(3) f = jnp.sin print(jax.vmap(apply_multi_grad, in_axes=(None, 0, 0))(f, order, x)) # [ 0.84147096 0.5403023 -0.84147096] # Compare by doing it manually: print(jnp.array([f(x[0]), jax.grad(f)(x[1]), jax.grad(jax.grad(f))(x[2])])) # [ 0.84147096 0.5403023 -0.84147096]
Question: I have the following pandas dataframe: import pandas as pd df = pd.DataFrame({'a': ["Q1", "Q2", "Q1", "P1"]})  That is:  a 0 Q1 1 Q2 2 Q1 3 P1  When I count the values with counts = df.value_counts(normalize=True), indexes change to tuple. That is, counts.index is now: MultiIndex([('Q1',), ('P1',), ('Q2',)], names=['a'])  but I want preserve the strings as indexes. In other words, I want counts.index to be an Index object instead of a MultiIndex. So, my questions are:  Why value_counts return a MultiIndex? Is there an alternative or a "fix" to that? 

Answer: You have a MultiIndex because you're starting from a DataFrame, not a Series. This is actually not very well described in the DataFrame.value_counts documentation:  The returned Series will have a MultiIndex with one level per input column but an Index (non-multi) for a single label.  What this actually means, is that unless you explicitly pass a single name to subset, this will create a MultiIndex. If you use subset, make sure to not wrap the name in a list (df.value_counts(subset=['a'], normalize=True) would still produce a MultiIndex). workaround If you don't want a MultiIndex, first slice the column or use the subset parameter: counts = df['a'].value_counts(normalize=True) # or counts = df.value_counts('a', normalize=True) counts.index # Index(['Q1', 'Q2', 'P1'], dtype='object', name='a')  Or squeeze to automatically convert a single column DataFrame to Series: counts = df.squeeze().value_counts(normalize=True) counts.index # Index(['Q1', 'Q2', 'P1'], dtype='object', name='a')
Question: I have a psycopg connection with autocommit turned on. Say, I run a query that is a combination of multiple queries, e.g.: query = ";".join([create_table, insert_data, analyze_table]) conn.execute(query)  Is this batch executed in a single transaction or multiple transactions? What happens if a query in the middle fails?

Answer: From Chain multiple statements within Psycopg2  In SQL chaining refers to the process of linking multiple SQL statements together into a single string, separated by semicolons. This allows you to execute multiple SQL statements at once, without having to execute them individually. For example, you might chain together a SELECT statement to retrieve data from a table, followed by an UPDATE statement to modify the data, and then a DELETE statement to remove it. When using chaining, it is important to note that each statement will be executed in the order they appear in the chain and that the results of one statement can be used in the next one. Additionally, when chaining SQL statements, if any statement in the chain fails, the entire chain will fail and none of the statements will be executed.
Question: I have 5 Azure Functions I'm developing, and 4 of them deploy and work fine, just having this trouble with one of them. I have the Azure Functions Core Tools installed and Azurite running as a VSCode plugin, and am able to run my function with no errors by calling func start --python --verbose from CLI. I can call the HTTP trigger from the local emulator and it returns a correct response. When I deploy to Azure Functions, the logs end in No HTTP triggers found with no errors shown throughout the deploy process. I've been able to download the web package via SCM and confirmed that the files are the same in there as they are in my development environment. What steps can I take to get more verbose information out of the Azure Functions environment to help with identifying the error which is causing the HTTP trigger to fail to deploy to production?

Answer: Turns out this was due to a custom environment variable missing from production which was present in development. I worked that one out on my own, I never found a useful error message to point to the fact that that is what was happening. If anyone knows how I'd be supposed to get a meaningful error message out of Azure Functions to point to such an error then please share, but I'll leave this answer here for anyone else going through the same thing, hopefully it helps you too.
Question: After sorting the data in pandas, I am saving it in Excel. At this time, we are trying to improve readability by changing the format. I specified the number format and background color at the same time, but there is an error where only one of the two is specified. I'm pretty sure this isn't an error, it's just 'I'm not understanding very well'. But I don't know where I made a mistake. I would like to receive your help. Below is the example code I wrote and the results of running it. The first code is when I formatted the number. It works normally. df = pd.DataFrame({ 'Name': ['John', 'Isac'], 'Value': [100000, 300000000] }) with pd.ExcelWriter('test.xlsx', engine='xlsxwriter') as writer: df.to_excel(writer, sheet_name='test', index=False) workbook = writer.book worksheet = writer.sheets['test'] format_with_commas = workbook.add_format({'num_format':'#,##0'}) worksheet.set_column('B:B', 15, format_with_commas)   However, the problem is that if you write code to specify the background color here, the number format will not work. I have included the code and execution results as images. df = pd.DataFrame({ 'Name': ['John', 'Isac'], 'Value': [100000, 300000000] }) styled_df = df.style.set_properties(**{'background-color':'yellow', 'color':'black'}) with pd.ExcelWriter('test.xlsx', engine='xlsxwriter') as writer: styled_df.to_excel(writer, sheet_name='test', index=False) workbook = writer.book worksheet = writer.sheets['test'] format_with_commas = workbook.add_format({'num_format':'#,##0'}) worksheet.set_column('B:B', 15, format_with_commas)   If my method is not correct, please let me know where I am wrong. Alternatively, if this code is outdated, please point that out as well.

Answer: I think you can use only pandas styles solution, but if check Styler.format:  Styler.format is ignored when using the output format Styler.to_excel, since Excel and Python have inherrently different formatting structures. However, it is possible to use the number-format pseudo CSS attribute to force Excel permissible formatting. See examples.  So try number-format pseudo CSS attribute, see last paragraph: styled_df = df.style.set_properties(**{'background-color':'yellow', 'color':'black'}) pseudo_css = 'number-format: #,##0' with pd.ExcelWriter('test.xlsx', engine='xlsxwriter') as writer: (styled_df.map(lambda v: pseudo_css, subset=['Value']) .to_excel(writer, sheet_name='test', index=False))  For oldier pandas versions: styled_df = df.style.set_properties(**{'background-color':'yellow', 'color':'black'}) pseudo_css = 'number-format: #,##0' with pd.ExcelWriter('test.xlsx', engine='xlsxwriter') as writer: (styled_df.applymap(lambda v: pseudo_css, subset=['Value']) .to_excel(writer, sheet_name='test', index=False))
Question: For example: T = TypeVar("T", bound="CustomDict") class CustomDict(dict): def __init__(self) -> None: super().__init__() class dict_operator: def __init__(self, method: Callable[..., Any]) -> None: self.method = method def __get__(self, instance: T, owner: Type[T]) -> Callable[..., Any]: def wrapper(key: Any, *args: Any, **kwargs: Any) -> Any: results = self.method(instance, key, *args, **kwargs) print("Did something after __getitem__ or __setitem__") return results return wrapper @dict_operator def __getitem__(self, key: Any) -> Any: return super().__getitem__(key) @dict_operator def __setitem__(self, key: Any, value: Any) -> None: super().__setitem__(key, value)  Mypy: Signature of "__getitem__" incompatible with supertype "dict"Mypyoverride Signature of "__getitem__" incompatible with supertype "Mapping"Mypyoverride (variable) Any: Any I take it the decorator has altered the overriden methods' signatures but I don't know how to account for this.

Answer: The issue isn't the type annotations on __getitem__ and __setitem__. For better or for worse, mypy (currently) doesn't recognise that a __get__ which returns a Callable is, for the majority of cases, a safe override in lieu of a real Callable. The fastest fix is to add a __call__ to your dict_operator class body (mypy Playground 1): class CustomDict(dict): ... class dict_operator: def __init__(self, method: Callable[..., Any]) -> None: ... def __get__(self, instance: T, owner: Type[T]) -> Callable[..., Any]: # This needs to be the same as the return type of `__get__` __call__: Callable[..., Any] @dict_operator def __getitem__(self, key: Any) -> Any: ... # OK @dict_operator def __setitem__(self, key: Any, value: Any) -> None: ... # OK   I don't know if you only did this for demonstration purposes, but IMO, you have too much imprecise typing here; Callable[..., Any] in particular is not a very useful type annotation. I don't know what Python version you're using, but if you're able to use typing_extensions instead, you can have access to the latest typing constructs for better static checking. See mypy Playground 2 for a possible way to implement stricter typing.