eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to get the following code to work: import polars as pl # Create sample DataFrames df1 = pl.DataFrame({ "Name": ["Alice", "Bob", "Charlie"], "Age": [25, 30, 35] }) df2 = pl.DataFrame({ "Product": ["Laptop", "Phone", "Tablet"], "Price": [1000, 500, 300] }) df3 = pl.DataFrame({ "City": ["New York", "San Francisco", "Chicago"], "Population": [8_400_000, 873_965, 2_746_388] }) def openwb(): return xlsxwriter.workbook("name.xlsx") def writewb(wb, df, sheetname): df.write_excel(workbook=wb, worksheet=sheetname) def main(): mywb = openwb() writewb(mywb, df1, "s1") writewb(mywb, df2, "s2") writewb(mywb, df3, "s3")  Problem is that each worksheet deletes the previously written ones, leaving me with only worksheet s3 in my workbook. Of course, this is oversimplified code. In reality, the functions above do a lot more stuff, and writing the worksheets is just one of the actions. Since the calls are spread across function calls, I am not using the "with workbook as wb ..." approach, since I feel that won't leave the workbook open across the function calls. How do I solve this? It seems I can convert the dataframe to pandas, but I am hoping for a polars-native solution

Answer: The workbook object needs to be closed. This is usually best achieved with a with context manager like in this polars/xlsxwriter example. However, you can also call an explicit close() on the workbook. Like this: import polars as pl import xlsxwriter # Create sample DataFrames df1 = pl.DataFrame({ "Name": ["Alice", "Bob", "Charlie"], "Age": [25, 30, 35] }) df2 = pl.DataFrame({ "Product": ["Laptop", "Phone", "Tablet"], "Price": [1000, 500, 300] }) df3 = pl.DataFrame({ "City": ["New York", "San Francisco", "Chicago"], "Population": [8_400_000, 873_965, 2_746_388] }) def openwb(): return xlsxwriter.Workbook("name.xlsx") def writewb(wb, df, sheetname): df.write_excel(workbook=wb, worksheet=sheetname) def main(): mywb = openwb() writewb(mywb, df1, "s1") writewb(mywb, df2, "s2") writewb(mywb, df3, "s3") mywb.close() if __name__ == "__main__": main()  Output:
Question: I want to make a heatmap in python (seaborn, matplotlib, etc) with two dimensions of information. I have a categorical value I want to assign to color, and a continuous variable (i.e. between 0-100 or 0-1) I want to assign to transparency, so each box has its own color and transparency (or intensity). for example: colors = pd.DataFrame([['b','g','r'],['black','orange','purple'],['r','yellow','white']]) transparency = pd.DataFrame([[0.1,0.2,0.3],[0.9,0.1,0.2],[0.1,0.6,0.3]])  how can I make a heatmap from this data such that the top left box is blue in color and 10% transparency (or 10% opaqueness, whichever), and so on? The best idea I have so far is to turn the colors into integer values, add those to the transparency values, and then make a custom colormap where each integer has a different color, ranging from white to the color in between the integer values. That sounds complicated to make and I'm hoping there's a built-in way to do this. Any ideas?

Answer: You could draw individual rectangles, giving each a specific color and transparency: import matplotlib.pyplot as plt from matplotlib.patches import Rectangle, Patch import pandas as pd colors = pd.DataFrame([['b', 'g', 'r'], ['black', 'orange', 'purple'], ['r', 'yellow', 'white']]) transparency = pd.DataFrame([[0.1, 0.2, 0.3], [0.9, 0.1, 0.2], [0.1, 0.6, 0.3]]) fig, ax = plt.subplots() for i, (color_col, transp_col) in enumerate(zip(colors.columns, transparency.columns)): for j, (color, transp) in enumerate(zip(colors[color_col], transparency[transp_col])): ax.add_patch(Rectangle((i - 0.5, j - 0.5), 1, 1, facecolor=color, alpha=transp, edgecolor='none', lw=0)) ax.invert_yaxis() # start at the top ax.autoscale(enable=True, tight=True) # recalculate axis limits ax.set_xticks(range(len(colors.columns)), colors.columns) ax.set_yticks(range(len(colors.index)), colors.index) plt.show()
Question: Suppose I have a bag of items {a, b}. Then I can choose pairs out of it in a variety of ways. One way might be to pick all possible permutations: [a, a], [a, b], [b, a], [b, b]. But I might disallow repetition, in which case the possible permutations are: [a, b], [b, a]. I might go further and declare that [a, b] is the same as [b, a], i.e. I only care about the "combination" of choices, not their permutations. For more about the distinction between combination vs. permutation, see: https://en.wikipedia.org/wiki/Combination What are the best ways to produce a combination of choices (i.e. order of elements should not matter)? My current solutions looks like this: import polars as pl choices = pl.DataFrame( [ pl.Series("flavor", ["x"] * 2 + ["y"] * 3), pl.Series("choice", ["a", "b"] + ["1", "2", "3"]), ] ) # join to produce the choices choices.join(choices, on=["flavor"]).with_columns( # generate a 2-element list representing the choice sorted_choice_pair=pl.concat_list("choice", "choice_right").list.sort() ).filter(pl.col.choice.eq(pl.col.sorted_choice_pair.list.first()))  shape: (9, 4) ┌────────┬────────┬──────────────┬────────────────────┐ │ flavor ┆ choice ┆ choice_right ┆ sorted_choice_pair │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ str ┆ str ┆ list[str] │ ╞════════╪════════╪══════════════╪════════════════════╡ │ x ┆ a ┆ a ┆ ["a", "a"] │ │ x ┆ a ┆ b ┆ ["a", "b"] │ │ x ┆ b ┆ b ┆ ["b", "b"] │ │ y ┆ 1 ┆ 1 ┆ ["1", "1"] │ │ y ┆ 1 ┆ 2 ┆ ["1", "2"] │ │ y ┆ 2 ┆ 2 ┆ ["2", "2"] │ │ y ┆ 1 ┆ 3 ┆ ["1", "3"] │ │ y ┆ 2 ┆ 3 ┆ ["2", "3"] │ │ y ┆ 3 ┆ 3 ┆ ["3", "3"] │ └────────┴────────┴──────────────┴────────────────────┘  So I generate all permutations, and then filter out those that where the "left element" does not match the first element of the list.

Answer: You can use .join_where() with a row index predicate to prevent "duplicates". (choices .with_row_index() .join_where(choices.with_row_index(), pl.col.flavor == pl.col.flavor_right, pl.col.index <= pl.col.index_right ) )  shape: (9, 5) ┌───────┬────────┬────────┬─────────────┬──────────────┐ │ index ┆ flavor ┆ choice ┆ index_right ┆ choice_right │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ u32 ┆ str ┆ str ┆ u32 ┆ str │ ╞═══════╪════════╪════════╪═════════════╪══════════════╡ │ 0 ┆ x ┆ a ┆ 0 ┆ a │ │ 0 ┆ x ┆ a ┆ 1 ┆ b │ │ 1 ┆ x ┆ b ┆ 1 ┆ b │ │ 2 ┆ y ┆ 1 ┆ 2 ┆ 1 │ │ 2 ┆ y ┆ 1 ┆ 3 ┆ 2 │ │ 3 ┆ y ┆ 2 ┆ 3 ┆ 2 │ │ 2 ┆ y ┆ 1 ┆ 4 ┆ 3 │ │ 3 ┆ y ┆ 2 ┆ 4 ┆ 3 │ │ 4 ┆ y ┆ 3 ┆ 4 ┆ 3 │ └───────┴────────┴────────┴─────────────┴──────────────┘
Question: Say we have an array like: a = np.array([ [k11, k12, k13, k14, k15, k16, k17, k18], [k21, k22, k23, k24, k25, k26, k27, k28], [k31, k32, k33, k34, k35, k36, k37, k38], [k41, k42, k43, k44, k45, k46, k47, k48] ]) const = C  I need to create a vector from this array like this (runge kutta 4): result = np.array([ const * (k11 + 2*k21 + 2*k31 + k41), const * (k12 + 2*k22 + 2*k32 + k42), const * (k13 + 2*k23 + 2*k33 + k43), .... const * (k18 + 2*k28 + 2*k38 + k48) ])  I am able to do this in cycle, but I am pretty sure numpy methods allow this in vectorised form.

Answer: np.einsum solution: result = const * np.einsum('ij,i->j', a, [1, 2, 2, 1])  ij,i are the dimensions of a and the coefficients. The result, j is missing i, which means that that dimension is multiplied and summed across the arrays. This solution is nice because it is very explicit about dimensions without requiring any reshaping or transposition. For larger matrices or longer multiplication chains, the order of operations will be optimized for speed.
Question: I am trying to import data from a .csv file into an Oracle DB using Python. So far it works fine if the .csv file contains 10 records. If I increase the number of records in the .csv file to 1.000.000, the script takes far too long and does not end even after an hour. Can anyone tell me how I can optimise my source code? King Regards Jegor ... sql_insert = """Merge into TEST_TABLE a Using (Select :ID as ID, :COUNTRY as COUNTRY , :DATE as DATE From Dual) src on src.ID = a.ID when matched then update set a.COUNTRY = src.COUNTRY, a.DATE = src.DATE when not matched then Insert (a.ID, a.COUNTRY, a.DATE) Values (src.ID, src.COUNTRY, src.DATE)""" # Get S3-File obj = s3.Object(CDH_S3_Bucket, CDH_Path + '/' + s3_filename) body = obj.get()['Body'].read().decode('utf-8').splitlines() # ---------------------------------------------------------- csv_reader = csv.reader(body, delimiter=',') headings = next(csv_reader) for line in csv_reader: data.append(line) if data: cursor.executemany(sql_insert, data) connection.commit() cursor.close() ... 

Answer: A merge is meant to modify one table based on the data in another table. It is not intended for single-row processing from the client like this. The proper design would be to use a normal bulk-bind insert to load a work table and then you can do a single merge execution to sync the target table with the work table. Also, when you do use a merge (appropriately), you don't want to use the where or delete subclauses within the merge when matched then update... clause. For merges, the where clause is different than in other SQL statements: it is a subprogram within the update program which means you already pay the penalty of CR block reads and other concurrency mechanisms, including redo, even if the where clause cancels the update of a row. You want to filter out unchanged rows within the using clause instead (which means pre-joining to the target table within the using clause and filter out unchanged rows there - you can then emit the target table's ROWID for optimal matching in the outer merge block; but the main benefit is filtering before the DML operation on the target). The delete subclause also fires after the update is processed and only on the updated row - if it's being updated, it's because it's coming in your new data and it won't be old by definition, unless you are getting old data in your files. To delete old records you have to use a totally separate archiving SQL that is not the same as your loading SQL.
Question: Am getting the following issue "pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 3036-12-31 00:00:00, at position 45100" I dont want to do the following as this will coerce all errors to NaT (not a time) s = pd.to_datetime(s, errors='coerce')  Is there no way to keep the dates and not convert them to NaT/nan/null? this is the function causing the error def remove_duplicates_based_on_keydate(df, id_col, date_col): df = df.copy() df.loc[:, date_col] = pd.to_datetime(df[date_col]) # Sort by the date column in descending order df_sorted = df.sort_values(by=date_col, ascending=False) # Drop duplicates, keeping the first occurrence (latest date) df_unique = df_sorted.drop_duplicates(subset=id_col, keep='first') # Sort again by id_col and reset index df_unique = df_unique.sort_values(by=id_col).reset_index(drop=True) return df_unique 

Answer: You can't have dates above Timestamp('2262-04-11 23:47:16.854775807'): Assuming dates like: df = pd.DataFrame({'date': ['2036-12-31 00:00:00', '3036-12-31 00:00:00']})  You could convert to periods with PeriodIndex df['periods'] = pd.PeriodIndex(df['date'].str.extract(r'^(\d{4}-\d\d-\d\d)', expand=False), freq='D')  Output:  date periods 0 2036-12-31 00:00:00 2036-12-31 1 3036-12-31 00:00:00 3036-12-31  Or for a precision to the second: df = pd.DataFrame({'date': ['2036-12-31 00:00:00', '3036-12-31 01:00:00']}) df['periods'] = pd.PeriodIndex(df['date'], freq='s')  Output:  date periods 0 2036-12-31 00:00:00 2036-12-31 00:00:00 1 3036-12-31 01:00:00 3036-12-31 01:00:00  Alternatively, first convert to datetime with errors='coerce', then to object and fillna with the missing dates: pd.to_datetime(df['date'], errors='coerce').astype(object).fillna(df['date'])  Note however that this will remain an object column with mixed strings and timestamps, which is not really useful in pandas. df['mixed'] = pd.to_datetime(df['date'], errors='coerce').astype(object).fillna(df['date']) print(df['mixed'].tolist()) # [Timestamp('2036-12-31 00:00:00'), '3036-12-31 00:00:00']
Question: I have created few Plotly charts with facets on basis of category variable and would like to change the color of facet text in the chart. Have searched alot even on plotly website but couldn't figure out the property that can be used to change the color for facet text. Using below image as an example I would like to change the color of - No & Yes: import plotly.express as px fig = px.scatter(px.data.tips(), x="total_bill", y="tip", facet_col="smoker") fig.for_each_annotation(lambda a: a.update(text=a.text.split("=")[-1])) fig.show()   Would really Appreciate any help !!

Answer: As for annotations, they are summarized in the layout attributes and can be done by making decisions based on the content of the text. In the following example, NO has been changed to red. import plotly.express as px fig = px.scatter(px.data.tips(), x="total_bill", y="tip", facet_col="smoker") fig.for_each_annotation(lambda a: a.update(text=a.text.split("=")[-1])) for t in fig.layout.annotations: if t.text.split("=")[-1] == 'No': t.font.color='red' fig.show()
Question: Take this MWE: from pympler import asizeof from random import randint, choice from string import printable from heapq import heappush ascii = printable[:-5] pq = [] for _ in range(10_000_000): heappush(pq, (randint(0, 31), randint(0, 31), randint(0, 31), ''.join(choice(ascii) for _ in range(16)))) print(asizeof.asizeof(pq))  I can see from running 'top' that this takes about 2.7GB of RAM. But asizeof reports 1,449,096,184 bytes which is a long way off. This is what 'top" shows:  /usr/bin/time -v gives: Maximum resident set size (kbytes): 2858616  Using another way of measuring RAM: from resource import getrusage, RUSAGE_SELF print(getrusage(RUSAGE_SELF).ru_maxrss * 1024)  This returns 2927054848 

Answer: asizeof rather accurately does what it's supposed to do: Measure the total size of the object structure. That's just not all the memory that Python uses. I get the exact same total 1,449,096,184 bytes with this minified test (Attempt This Online!): from sys import getsizeof def size(obj, align=8): return getsizeof(obj) // -align * -align a = [] for _ in range(10_000_000): a.append((0, 0, 0, ' ' * 16)) list_size = size(a) tuple_size = size(a[0]) str_size = size(a[0][3]) ints_size = sum(map(size, range(32))) print(f'{ list_size + len(a) * (tuple_size + str_size) + ints_size :,}')  Using align=16 (also in asizeof) would be more realistic, that's likely the alignment your Python uses. I get 1,529,096,192 bytes then.
Question: The easiest way is to demonstrate my question with an example. Suppose I have the following long format data frame In [284]: import pandas as pd In [285]: data = pd.DataFrame({"day": [0,0,0,0,0,0,1,1,1,1,1,1], "cat1": ["A", "A", "A", "B", "B", "B", "A", "A", "B", "B", "B", "B"], "cat2":["1", "1", "2", "1", "2", "2", "1", "2", "1", "1", "2", "2"], "value": [10, 230, 32,12, 12, 65, 12, 34, 97, 0, 12,1]}) In [286]: data Out[286]: day cat1 cat2 value 0 0 A 1 10 1 0 A 1 230 2 0 A 2 32 3 0 B 1 12 4 0 B 2 12 5 0 B 2 65 6 1 A 1 12 7 1 A 2 34 8 1 B 1 97 9 1 B 1 0 10 1 B 2 12 11 1 B 2 1  Per day I have two categories. My goal is to aggregate the cat2 category in a specific way. For each tuple (date, cat1, cat2) I would like to perform the following: In [287]: data_day = data[data["day"]==0] In [288]: data_day_cat1 = data_day[data_day["cat1"]=="A"] In [289]: data_day_cat1_cat2 = data_day_cat1[data_day_cat1["cat2"]=="1"] In [290]: data_day_cat1_cat2["value"].pow(2).mean() Out[290]: np.float64(26500.0) In [291]: data_day_cat1_cat2 = data_day_cat1[data_day_cat1["cat2"]=="2"] In [292]: data_day_cat1_cat2["value"].pow(2).mean() Out[292]: np.float64(1024.0)  That is on the first day, for cat1 being A, I want a single line for all occurrence of cat2, where the latter is like a "root mean square error". Currently I'm looping over all combination, but I was playing around with using groupby. However, something like: data.groupby(["day", "cat1", "cat2"])["value"].apply(lambda x: x**2).mean()  Does work. What I would like to get is a DataFrame like this:  day cat1 cat2 value 0 0 A 1 26500 1 0 A 2 1024  EDIT: Note, I want the complete DataFrame, was just too lazy to write down the whole data frame. Is this possible without looping over all day, cat1 and cat2? Could groupby be used?

Answer: You can create a new column with the square value and then do the groupby: data["value2"] = data["value"] * data["value"] gb = data.groupby(["day", "cat1", "cat2"])["value2"].mean() display(gb)
Question: Getting this typing error: error: Incompatible types in assignment (expression has type "object", variable has type "A | B") [assignment]  With this code: from dataclasses import dataclass from typing import TypeVar, Mapping, reveal_type @dataclass class A: foo: str = "a" @dataclass class B: bar: str = "b" lookup_table: Mapping[str, type[A] | type[B]] = { "a": A, "b": B } reveal_type(lookup_table) # note: Revealed type is "typing.Mapping[builtins.str, Union[type[simple.A], type[simple.B]]]" T = TypeVar("T") def load(lookup_table: Mapping[str, type[T]], lookup_key:str) -> T: con: type[T] = lookup_table[lookup_key] instance: T = con() return instance example_a: A | B = load(lookup_table, "a") # error: Incompatible types in assignment (expression has type "object", variable has type "A | B") print(example_a)   Edit: Logged a mypy bug here: https://github.com/python/mypy/issues/18265

Answer: This is a mypy bug present in 1.13.0 and below (previously reported here and by OP here). pyright and basedmypy both accept the given snippet. mypy stores type[A | B] types as a union of types internally (type[A] | type[B]). This is usually convenient, but causes troubles when solving type[T] <: type[A] | type[B] for T, because most types aren't "distributive" (P[A, B] is not equivalent to P[A] | P[B]), and type special case isn't taken into account yet. General solver produces meet(A, B) in such case by solving type[T] <: type[A] => T <: A | | => T <: meet(A, B) type[T] <: type[B] => T <: B |  When A and B have no explicit parent in common, the closest supertype is object. To understand the logic behind that, consider similar equations: x < 2 && x < 3 => x < min(2, 3), and intersection/meet is for types what min operation is for numbers (and union/join is similar to max). I submitted a PR to special-case this behaviour, so things may change in a future mypy release.
Question: I have a project I'm working on for a data analysis course, where we pick a data set and go through the steps of cleaning and exploring the data with a question to answer in mind. I want to be able to see how many instances of the data occur in different years, but right now the Year column in the data set is set to a datatype object, with values spanning from whole years like 1998, just the last 2 digits like 87, ranges of presumed years ('early 1990's', '89 or 90', '2011- 2012', 'approx 2001'). I'm trying to determine the best way to convert all these various instances to the proper format or would it be better to drop the values that are not definitive? I worry that this would lead to too much data loss because the dataset is already pretty small (about 5000 rows total). I have looked into regex and it seems like that is the path I should go down to keep and alter the values, but I still don't understand it conceptually very well, and I worry about the efficiency of filtering for so many different value variations. I'm still very new to Python and pandas.

Answer: Assuming your Year columns are strings, I would write a normalize function like this: import re import pandas as pd data = [ {"year": "early 1990's"}, {"year": "89 or 90"}, {"year": "2011-2012"}, {"year": "approx 2001"}, ] def normalize(row): year = row["year"] # Count the number of digits count = len(re.findall("\\d", year)) if count == 4: # match YYYY if m := re.search("\\d\\d\\d\\d", year): return m.group(0) if count == 2: # match YY if m := re.search("\\d\\d", year): return "19" + m.group(0) df = pd.DataFrame(data) df["normalized"] = df.apply(normalize, axis=1) print(df) => year normalized 0 early 1990's 1990 1 89 or 90 None 2 2011-2012 None 3 approx 2001 2001  The function returns None for unmatched pattern. You can list them as follows: >>> print(df[df["normalized"].isnull()]) ... year normalized 1 89 or 90 None 2 2011-2012 None  Review the output and modify the normalize function as you like. Repeat these steps until you get satisfied.
Question: I want to have horizontally aligned 3D and 2D plots, where the y-axis of the 2D plot is the same height as the z-axis of the 3D plot. The following code produces a default output: import matplotlib.pyplot as plt fig = plt.figure() fig.set_size_inches(10, 4) fig.subplots_adjust(wspace=0.5) # 3D surface plot ax1 = fig.add_subplot(121, projection="3d") ax1.set(zlabel="Value") # 2D line plot ax2 = fig.add_subplot(122) ax2.set(ylabel="Value") plt.show()   I want to shrink down the right hand side 2D subplot so that the y-axis ("Value") is a similar height to (or preferably the same height as) the z-axis ("Value") on the 3D plot (ideally I want them to line up). I haven't been able to find a way to change subplot sizes independently like this in matplotlib (setting relative ratios and gridspec don't seem able to achieve this). Any help would be much appreciated.

Answer: You can use set_position() to change the dimensions of one of the subplot: plt.figure(1).axes[1].set_position([0.6,0.4,0.25,0.3]) # left, bottom, width, height  It gives:
Question: I have a 2-dimensional array that represents a grid. The (numpy) array is as follows: dx_grid =[[ "A", "B", "C"], [ "L", "M", "N"], [ "X", "Y", "Z"]]  I want to convert that into the following:  I know that grid_2d_graph can connect 4 adjacent nodes. For example, it would connect node M to B, L, N and Y BUT NOT to A, C, X, Z. How would I create such a graph using networkx in python?

Answer: Something like this should work: import networkx as nx dx_grid =[[ "A", "B", "C"], [ "L", "M", "N"], [ "X", "Y", "Z"]] r, c = len(dx_grid), len(dx_grid[0]) g = nx.Graph() # add nodes for i in range(r): for j in range(c): g.add_node(dx_grid[i][j], pos=(j,r-i)) # add edges # for all nodes for i in range(r): for j in range(c): # connect all neighbors for k in range(-1,2): for l in range(-1,2): # check if neighbor node index is valid if i+k >= 0 and i+k < r and j+l >= 0 and j+l < c: if k == 0 and l == 0: continue # avoid self-loops g.add_edge(dx_grid[i][j], dx_grid[i+k][j+l]) # connect with neighbor node # get positions pos = nx.get_node_attributes(g,'pos') # draw network with nodes at given positions plt.figure(figsize=(6,6)) nx.draw(g, pos=pos, node_color='lightblue', with_labels=True, node_size=600)