eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm trying to combine multiple columns into 1 column with python Polars. However I don't seem to find an (elegant) way to combine columns into a list. I only need to combine column b - e into 1 column. Column a needs to stay exactly as it is now. I've tried using map_elements to achieve this. Despite the fact that this isn't working its also slow and must likely not the best way to do this. Anyone who can help me out with how I can achieve this result? The dataframe I have: df = pl.from_repr(""" ┌─────┬─────┬─────┬─────┬─────┐ │ a ┆ b ┆ c ┆ d ┆ e │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═════╪═════╪═════╪═════╪═════╡ │ 0.1 ┆ 1.1 ┆ 2.1 ┆ 3.1 ┆ 4.1 │ │ 0.2 ┆ 1.2 ┆ 2.2 ┆ 3.2 ┆ 4.2 │ │ 0.3 ┆ 1.3 ┆ 2.3 ┆ 3.3 ┆ 4.3 │ └─────┴─────┴─────┴─────┴─────┘ """)  The result I need: shape: (3, 2) ┌─────┬──────────────────────┐ │ a ┆ value │ │ --- ┆ --- │ │ f64 ┆ list[f64] │ ╞═════╪══════════════════════╡ │ 0.1 ┆ [1.1, 2.1, 3.1, 4.1] │ │ 0.2 ┆ [1.2, 2.2, 3.2, 4.2] │ │ 0.3 ┆ [1.3, 2.3, 3.3, 4.3] │ └─────┴──────────────────────┘ 

Answer: Use concat_list. pl.Config(fmt_table_cell_list_len=10, fmt_str_lengths=80) # increase repr defaults  df.select('a', value=pl.concat_list(pl.exclude('a')))  shape: (3, 2) ┌─────┬──────────────────────┐ │ a ┆ value │ │ --- ┆ --- │ │ f64 ┆ list[f64] │ ╞═════╪══════════════════════╡ │ 0.1 ┆ [1.1, 2.1, 3.1, 4.1] │ │ 0.2 ┆ [1.2, 2.2, 3.2, 4.2] │ │ 0.3 ┆ [1.3, 2.3, 3.3, 4.3] │ └─────┴──────────────────────┘
Question: I would like to know how to fill a column of a polars dataframe with random values. The idea is that I have a dataframe with a given number of columns, and I want to add a column to this dataframe which is filled with different random values (obtained from a random.random() function for example). This is what I tried for now: df = df.with_columns( pl.when((pl.col('Q') > 0)).then(random.random()).otherwise(pl.lit(1)).alias('Prob') )  With this method, the result that I obtain is a column filled with one random value i.e. all the rows have the same value. Is there a way to fill the column with different random values ? Thanks by advance.

Answer: You need a "column" of random numbers the same height as your dataframe? np.random.rand could be useful for this: df = pl.DataFrame({"foo": [1, 2, 3]}) df.with_columns(pl.lit(np.random.rand(df.height)).alias("prob"))  shape: (3, 2) ┌─────┬──────────┐ │ foo ┆ prob │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞═════╪══════════╡ │ 1 ┆ 0.657389 │ │ 2 ┆ 0.616265 │ │ 3 ┆ 0.142611 │ └─────┴──────────┘  df.with_columns( pl.when(pl.col("foo") > 2).then(pl.lit(np.random.rand(df.height))) .alias("prob") )  shape: (3, 2) ┌─────┬──────────┐ │ foo ┆ prob │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞═════╪══════════╡ │ 1 ┆ null │ │ 2 ┆ null │ │ 3 ┆ 0.686551 │ └─────┴──────────┘  It may also be possible to do similar with expressions? e.g. with .int_range() and .sample() df.with_columns( (pl.int_range(1000).sample(pl.len(), with_replacement=True) / 1000) .alias("prob") )  shape: (3, 2) ┌─────┬───────┐ │ foo ┆ prob │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞═════╪═══════╡ │ 1 ┆ 0.288 │ │ 2 ┆ 0.962 │ │ 3 ┆ 0.734 │ └─────┴───────┘
Question: I am very new to SHAP and I would like to give it a try, but I am having some difficulty. The model is already trained and seems to perform well. I then use the training data to test SHAP with. It looks like so:  var_Braeburn var_Cripps Pink var_Dazzle var_Fuji var_Granny Smith \ 0 1 0 0 0 0 1 0 1 0 0 0 2 0 1 0 0 0 3 0 1 0 0 0 4 0 1 0 0 0 var_Other Variety var_Royal Gala (Tenroy) root_CG202 root_M793 \ 0 0 0 0 0 1 0 0 1 0 2 0 0 1 0 3 0 0 0 0 4 0 0 0 0 root_MM106 ... frt_BioRich Organic Compost_single \ 0 1 ... 0 1 0 ... 0 2 0 ... 0 3 1 ... 0 4 1 ... 0 frt_Biomin Boron_single frt_Biomin Zinc_single \ 0 0 1 1 0 0 2 0 0 3 0 0 4 0 0 frt_Fertco Brimstone90 sulphur_single frt_Fertco Guano _single \ 0 0 0 1 0 0 2 0 0 3 0 0 4 0 0 frt_Gro Mn_multiple frt_Gro Mn_single frt_Organic Mag Super_multiple \ 0 0 0 0 1 1 0 1 2 1 0 1 3 1 0 1 4 1 0 1 frt_Organic Mag Super_single frt_Other Fertiliser 0 0 0 1 0 0 2 0 0 3 0 0 4 0 0  I then do explainer = shap.Explainer(model) and shap_values = explainer(X_train) This runs without error and shap_values gives me this: .values = array([[[ 0.00775555, -0.00775555], [-0.03221035, 0.03221035], [-0.0027203 , 0.0027203 ], ..., [ 0.00259787, -0.00259787], [-0.00459262, 0.00459262], [-0.0303394 , 0.0303394 ]], [[-0.00068313, 0.00068313], [-0.03006355, 0.03006355], [-0.00245706, 0.00245706], ..., [-0.00418809, 0.00418809], [-0.00088372, 0.00088372], [-0.00030019, 0.00030019]], [[-0.00068313, 0.00068313], [-0.03006355, 0.03006355], [-0.00245706, 0.00245706], ..., [-0.00418809, 0.00418809], [-0.00088372, 0.00088372], [-0.00030019, 0.00030019]], ...,  However, when I then run shap.plots.beeswarm(shap_values), I get the following error: ValueError: The beeswarm plot does not support plotting explanations with instances that have more than one dimension! What am I doing wrong here?

Answer: Try this: from shap import Explainer from shap.plots import beeswarm from sklearn.ensemble import RandomForestClassifier from sklearn.datasets import load_breast_cancer X, y = load_breast_cancer(return_X_y=True, as_frame = True) model = RandomForestClassifier().fit(X, y) explainer = Explainer(model) sv = explainer(X)  Then, because RF is a bit special, retrieve shap values just for class 1: beeswarm(sv[:,:,1])  EDIT: Somehow I came to my own post one year later after receiving: ValueError: The beeswarm plot does not support plotting explanations with instances that have more than one dimension!  So, instead of stating "RF is special", I have to state: "With every model you receive this kind of error message, try reducing dimentionality like sv[:,:,class_index]"
Question: I have a typing.ForwardRef object as a remnant from earlier generic programming shenanigans. At this point I know the class represented by the ForwardRef exists, but how can I retrieve this type? A fairly minimal example of what I am doing. Convoluted solution for this example, but for the actual use case it makes sense. class GenericClass(Generic[T]): def __init_subclass__(cls, /, **kwargs) -> None: # retrieve the type or forwardref of T orig_bases = [orig_base for orig_base in cls.__orig_bases__ if get_origin(orig_base) is GenericClass] assert len(orig_bases) == 1 orig_base = orig_bases[0] generic_types = get_args(orig_base) assert len(generic_types) == 1 cls.__type_t = generic_types[0] def use_type(self) -> T: thetype = resolve_forward_refs(self.__type_t) # to be implemented return thetype() class MyImpl(GenericClass["SomeType"]): pass 

Answer: You can evaluate a ForwardRef by calling ForwardRef._evaluate(globals(), locals(), frozenset()) on it. This assumes that the type behind the forward ref is known in these namespaces, i.e. you have imported the type into the local or global scope of the module. if isinstance(myref, ForwardRef): myref_type = myref._evaluate(globals(), locals(), frozenset())  Alternative syntax: typing._eval_type(myref, globals(), locals())  Source: https://github.com/python/cpython/blob/v3.11.3/Lib/typing.py#L849 Edit: For Python versions <3.9, leave out the third parameter to _evaluate: if isinstance(myref, ForwardRef): myref_type = myref._evaluate(globals(), locals())
Question: I am trying to run the below script from plotly: https://plotly.com/python/county-choropleth/ I'm receiving the error code right out the gate: TypeError: 'MultiPolygon' object is not iterable I've looked up several posts where this is a similar issue, but I'm skeptical these are solutions for this particular issue. OPTION 2 seems like the more likely approach, but why would there be a workaround for simple coding that plotly is publishing? Seems I might be missing something in the way the code is written. OPTION 1: 'Polygon' object is not iterable- iPython Cookbook OPTION 2: Python: Iteration over Polygon in Dataframe from Shapefile to color cartopy map import plotly.figure_factory as ff import numpy as np import pandas as pd df_sample = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/minoritymajority.csv') df_sample_r = df_sample[df_sample['STNAME'] == 'Florida'] values = df_sample_r['TOT_POP'].tolist() fips = df_sample_r['FIPS'].tolist() endpts = list(np.mgrid[min(values):max(values):4j]) colorscale = ["#030512","#1d1d3b","#323268","#3d4b94","#3e6ab0", "#4989bc","#60a7c7","#85c5d3","#b7e0e4","#eafcfd"] fig = ff.create_choropleth( fips=fips, values=values, scope=['Florida'], show_state_data=True, colorscale=colorscale, binning_endpoints=endpts, round_legend_values=True, plot_bgcolor='rgb(229,229,229)', paper_bgcolor='rgb(229,229,229)', legend_title='Population by County', county_outline={'color': 'rgb(255,255,255)', 'width': 0.5}, exponent_format=True, ) fig.layout.template = None fig.show() 

Answer: I had the same error message recently and came across this post. I think the key is the migration of shapely to 2.x. You need to use the ".geoms" attribute now to make multi-part geometries iterable: https://shapely.readthedocs.io/en/stable/migration.html This resolved my issue.
Question: I'm currently working in a project in which I'm using Flask and Yolov8 together. When I run this code from ultralytics import YOLO model = YOLO("./yolov8n.pt") results = model.predict(source="../TEST/doggy.jpg", save=True, save_txt=True)  the output will be saved in this default directory /run/detect/ like Ultralytics YOLOv8.0.9 Python-3.10.8 torch-2.0.0+cpu CPU Fusing layers... YOLOv8n summary: 168 layers, 3151904 parameters, 0 gradients, 8.7 GFLOPs Results saved to d:\runs\detect\predict4 1 labels saved to d:\runs\detect\predict4\labels  and what I want is the predict directory number or the entire directory path in a variable. I tried capturing the path using sys.stdout methods but i want a direct solution.

Answer: You can change the directory where the results are saved (save_dir) by modifying two arguments in predict: project and name results = model.predict(source=xxx, save_txt = True, project="xxx", name="yyy")  such that: save_dir=project/name
Question: In Pandas, one can perform boolean operations on boolean DataFrames with the all and any methods, providing an axis argument. For example: import pandas as pd data = dict(A=["a","b","?"], B=["d","?","f"]) pd_df = pd.DataFrame(data)  For example, to get a boolean mask on columns containing the element "?": (pd_df == "?").any(axis=0)  and to get a mask on rows: (pd_df == "?").any(axis=1)  Also, to get a single boolean: (pd_df == "?").any().any()  In comparison, in polars, the best I could come up with are the following: import polars as pl pl_df = pl.DataFrame(data)  To get a mask on columns: (pl_df == "?").select(pl.all().any())  To get a mask on rows: pl_df.select( pl.concat_list(pl.all() == "?").alias("mask") ).select( pl.col("mask").list.eval(pl.element().any()).list.first() )  And to get a single boolean value: pl_df.select( pl.concat_list(pl.all() == "?").alias("mask") ).select( pl.col("mask").list.eval(pl.element().any()).list.first() )["mask"].any()  The last two cases seem particularly verbose and convoluted for such a straightforward task, so I'm wondering whether there are shorter/faster equivalents?

Answer: I think one thing that might be making this more confusing is that you're not doing everything in the select context. In other words, don't do this: (pl_df == "?") The first thing we can do is pl_df.select(pl.all()=="?") shape: (3, 2) ┌───────┬───────┐ │ A ┆ B │ │ --- ┆ --- │ │ bool ┆ bool │ ╞═══════╪═══════╡ │ false ┆ false │ │ false ┆ true │ │ true ┆ false │ └───────┴───────┘  When we call pl.all() it means all of the columns. For each column we're converting its original value into a bool of whether or not it's equal to ? Now let's do this: pl_df.select((pl.all()=="?").any()) shape: (1, 2) ┌──────┬──────┐ │ A ┆ B │ │ --- ┆ --- │ │ bool ┆ bool │ ╞══════╪══════╡ │ true ┆ true │ └──────┴──────┘  This gives you the per column. All we did was add .any which tells it that if anything in the parenthesis that preceded it is true then return True. Now let's do pl_df.select(pl.any_horizontal(pl.all()=="?")) shape: (3, 1) ┌───────┐ │ any │ │ --- │ │ bool │ ╞═══════╡ │ false │ │ true │ │ true │ └───────┘  When we call pl.any_horizontal(...) then it is going to do that rowwise for whatever ... is. Lastly, if we put them together... pl_df.select(pl.any_horizontal(pl.all()=="?").any()) shape: (1, 1) ┌──────┐ │ any │ │ --- │ │ bool │ ╞══════╡ │ true │ └──────┘  then we get the single value indicating that somewhere in the dataframe is an item that is equal to "?"
Question: python 3.9.2-3 pandas 2.0.0 pandas-io 0.0.1  Error: from pandas.io.json import json_normalize ImportError: cannot import name 'json_normalize' from 'pandas.io.json' (/home/casaos/.local/lib/python3.9/site-packages/pandas/io/json/__init__.py)  Apparently this was a problem early on in the pre 1x days of pandas, but seems to have resurfaced. Suggestions? I'm running a script which was functional previously, but migrating it to a new host. It errors out on the line: from pandas.io.json import json_normalize  and throws the error ImportError: cannot import name 'json_normalize' from 'pandas.io.json' (/home/casaos/.local/lib/python3.9/site-packages/pandas/io/json/__init__.py)  I've attempted to reinstall pandas ('install' option), remove and reinstall, and 'install --force-reinstall' all performed as root so that the base install of python3 has it installed as opposed to a single user

Answer: This was indeed the solution simply removing the import line. I'd have liked an attribute to check to determine the pandas version installed easily, but have to settle for a try:except: to determine if the import is needed. pandas.io.json.json_normalize was deprecated and removed in the newest version. Use pandas.json_normalize. Also, the tutorial you were following is most probably severely outdated. You are on your own now. – Ξένη Γήινος Apr 23 at 7:48
Question: url = 'https://query2.finance.yahoo.com/v7/finance/quote?symbols=TSLA&fields=regularMarketPreviousClose&region=US&lang=en-US' headers = { 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36' } data = requests.get(url,headers=headers) prepost_data = data.json()  It seems recently, Yahoo Finance changed their V7 API to require cookies for every request. Running the code above, I get the Invalid Crumb error {"finance":{"result":null,"error":{"code":"Unauthorized","description":"Invalid Crumb"}}} This issue seems to also be known in this Github repo: https://github.com/joshuaulrich/quantmod/issues/382 They seem to have a patch that works: https://github.com/joshuaulrich/quantmod/pull/383/commits But the code is all written in R... Anyone know how to translate this to Python?

Answer: This worked for me.  Make HTTP GET call to URL https://fc.yahoo.com. Although this call result in a 404 error, we just need it to extract set-cookie from response headers which is then used in the subsequent calls Now make an HTTP GET call to the URL https://query2.finance.yahoo.com/v1/test/getcrumb, by including the obtained cookie from the previous response headers. This call will retrieve the crumb value. Replace [crumb-value] in the following URL and make a HTTP GET call with cookie https://query2.finance.yahoo.com/v7/finance/quote?symbols=TSLA&crumb=[crumb-value] Cache the cookie value and crumb value to skip first two steps going forward
Question: From what I read, to make sure that the typing information of your code is distributed alongside your code for linters to read, the py.typed file should be part of your distribution. I find answers for how to add these to setup.py but it is not clear to me 1. whether it should be included in pyproject.toml (using setuptools), 2. if so, how it should be added. Scouring their github repository, it seems that this is not added automatically so the question remains how I should add it to my pyproject.toml. I found this general discussion about package_data but it includes reference to include_package_data and a MANIFEST.in and it gets confusing from there what should go where. Tl;dr: how should I include py.typed in pyproject.toml when using setuptools?

Answer: Yes, you should add py.typed to your package's source folder (same level as __init__.py). This informs type checkers, like mypy, that your package supports typing. See PEP 561. Here is an example of what is needed in pyproject.toml. Replace pkgname with the name of your package. [tool.setuptools.package-data] "pkgname" = ["py.typed"]  It is worth noting package discovery using: [tool.setuptools.packages.find] where = ["src"]  seems to be required, alas package-data wouldn't have any effect, as explained in setuptools docs: https://setuptools.pypa.io/en/latest/userguide/datafiles.html#package-data
Question: I create my pyspark dataframe: from pyspark.sql import SparkSession from pyspark.sql.types import StructType, StructField, BinaryType, ArrayType, StringType, TimestampType input_schema = StructType([ StructField("key", StringType()), StructField("headers", ArrayType( StructType([ StructField("key", StringType()), StructField("value", StringType()) ]) )), StructField("timestamp", TimestampType()) ]) input_data = [ ("key1", [{"key": "header1", "value": "value1"}], datetime(2023, 1, 1, 0, 0, 0)), ("key2", [{"key": "header2", "value": "value2"}], datetime(2023, 1, 1, 0, 0, 0)), ("key3", [{"key": "header3", "value": "value3"}], datetime(2023, 1, 1, 0, 0, 0)) ] df = spark.createDataFrame(input_data, input_schema)  I want to use Pandas' assert_frame_equal(), so I want to convert my dataframe to a Pandas dataframe. df.toPandas() will throw TypeError: Casting to unit-less dtype 'datetime64' is not supported. Pass e.g. 'datetime64[ns]' instead. How can I successfully convert the "timestamp" column in order to not lose detail of the datetime value? I need them to remain to 2023-01-01 00:00:00 and not 2023-01-01.

Answer: I found the solution: from pyspark.sql.functions import date_format df = df.withColumn("timestamp", date_format("timestamp", "yyyy-MM-dd HH:mm:ss")).toPandas()  Now I was able to use assert_frame_equal(df, test_df)  successfully. It did not lose precision.
Question: I need to change a text color of a code cell (namely code cell, not just a text block) that is defined using "`" symbols in Markdown. All my attempts have failed and the color stays the same, which is by default black I have tried to use <font color="...">...<\font> attribute and also <span style="color:...;">...<\span> attribute. All of them didn't work and the text is still the same as on the attached image. How can I solve this problem and finally change the color to whatever I want? Code cell which text I want to change

Answer: If you want to add syntax highlighting to a programming language, it can be written like this: # For example, you want to use Python ```python print("the syntax is highlighted") ```  The result would be like this: print("the syntax is highlighted")  EDIT: As far as I know, Markdown doesn't support color. So what you'll need is wrap the code cell as HTML. Instead of this: `some beautiful text here`  You would want to write this: <code>some beautiful text here</code>  And then add the span tag to change the color: <code>some <span style='color:blue'>beautiful</span> text here</code>  At this point, the word 'beautiful' will change to blue. It works on my VSCode Markdown viewer. But different app will render Markdown differently.
Question: I was trying to implement a conjugate gradient algorithm using Dask (for didactic purposes) when I realized that the performance were way worst that a simple numpy implementation. After a few experiments, I have been able to reduce the problem to the following snippet: import numpy as np import dask.array as da from time import time def test_operator(f, test_vector, library=np): for n in (10, 20, 30): v = test_vector() start_time = time() for i in range(n): v = f(v) k = library.linalg.norm(v) try: k = k.compute() except AttributeError: pass print(k) end_time = time() print('Time for {} iterations: {}'.format(n, end_time - start_time)) print('NUMPY!') test_operator( lambda x: x + x, lambda: np.random.rand(4_000, 4_000) ) print('DASK!') test_operator( lambda x: x + x, lambda: da.from_array(np.random.rand(4_000, 4_000), chunks=(2_000, 2_000)), da )  In the code, I simply multiply by 2 a vector (this is what f does) and print its norm. When running with dask, each iteration slows down a little bit more. This problem does not happen if I do not compute k, the norm of v. Unfortunately, in my case, that k is the norm of the residual that I use to stop the conjugate gradient algorithm. How can I avoid this problem? And why does it happen? Thank you!

Answer: I think the code snippet is missusing lazy evaluation in dask, specifically the addition operation. Without optimization, the addition lambda x: x+x is complicating the execution graph, with the depth growing with counter, hence overheads. More precisely, for the counter value i we handle the graph of O(i) when computing the norm, so that the total runtime is O(n**2). Of course optimization is possible and desired, but I stop here as the example shared is synthetic. Below I demonstrate that the graph grows linearly with the counter.  To see the quadratic complexity visually, consider the following cleaned version of the snippet in question import numpy as np import dask.array as da from time import time import matplotlib.pyplot as plt ns = (10, 20, 40, 50, 60) def test_operator(f, v, norm): out = [] for n in ns: start_time = time() for i in range(n): v = f(v) norm(v) end_time = time() out.append(end_time - start_time) return out out = test_operator( lambda x:x+x, np.random.rand(4_000, 4_000), norm = np.linalg.norm ) plt.scatter(ns,out,label='numpy') out = test_operator( lambda x:x+x, da.from_array(np.random.rand(4_000, 4_000), chunks=(2_000, 2_000)), norm = lambda v: da.linalg.norm(v).compute() ) plt.scatter(ns,out,label='dask') plt.legend() plt.show()
Question: In the example below, how can I change the background color of "Title" to be red (with alpha=.5), not the font color of "Title"? import matplotlib.pyplot as plt fig, ax = plt.subplots() ax.plot([1,2,3,], [1,2,3], label='Series') legend = ax.legend(title='Title') # Access to the legend title box adapted from # https://stackoverflow.com/a/63570572 legend._legend_title_box._text.set_color('red')  

Answer: You can change the color of the bounding box of the title text: import matplotlib.pyplot as plt fig, ax = plt.subplots() ax.plot([1, 2, 3, ], [1, 2, 3], label='Series') legend = ax.legend(title='Title') legend._legend_title_box._text.set_bbox(dict(facecolor='red', alpha=0.3, lw=0)) plt.show()   PS: For the red background to span the entire width of the legend frame, you could pad the title with whitespace, as proposed by Trenton Mckinney. E.g. legend = ax.legend(title=f'{"Title": ^15}', frameon=False)
Question: Python has a defaultdict class which behaves like a dictionary, except that it outputs a default value when encountering a new key. This default value is created using the default_factory attribute, which is a function that takes no arguments. For example, d = defaultdict(int) creates a dictionary d whose default value is 0, since int() returns 0. Why is the default value stored as a function which takes no arguments, rather than storing the default value itself? I am assuming there is a reason behind this choice of implementation, but I don't see what it is. From a naive perspective, it seems like storing the default value would be just as effective and less convoluted.

Answer: Think about how this would work for mutable types. For example, d = defaultdict(list) creates a new list object for each unassigned key that is requested. If you used a single instance, then each key would be returning the same instance of that list. Appending to an instance returned by one key, e.g. d[1].append(2) would have the observable behavior of appending the value to every other key in the dict. That behavior in almost all cases would be undesirable. Yes, for primitive/immutable datatypes, like int, float, str, etc, a default value would be acceptable, but that does not cover a huge use-case for containers or other mutable structures where you would want the value for each key to be unique.