eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to extend polars API as described in the docs, like this: @pl.api.register_expr_namespace("greetings") class Greetings: def __init__(self, expr: pl.Expr): self._expr = expr def hello(self) -> pl.Expr: return (pl.lit("Hello ") + self._expr).alias("hi there") def goodbye(self) -> pl.Expr: return (pl.lit("Sayōnara ") + self._expr).alias("bye")  If I were to put the actual registration in a dedicated module (extensions.py), how am I supposed to import the methods from the respective class from within another module? Going with the dataframe example in the docs, let's say I put the following code in a module called target.py. I need to make the greetings-namespace available. How can I do it, i.e. how excactly should the import look like? pl.DataFrame(data=["world", "world!", "world!!"]).select( [ pl.all().greetings.hello(), pl.all().greetings.goodbye(), ] ) 

Answer: You could just import extensions(assuming that the file is relative) in your target.py file to register the greetings namespace. import polars as pl import extensions # noqa: F401 pl.DataFrame(data=["world", "world!", "world!!"]).select( [ pl.all().greetings.hello(), pl.all().greetings.goodbye(), ] )  See also:  polars-xdt
Question: Suppose we create some dates: import polars as pl df = pl.DataFrame( [ pl.Series("start", ["2023-01-01"], dtype=pl.Date).str.to_date(), pl.Series("end", ["2024-01-01"], dtype=pl.Date).str.to_date(), ] )  Now I can create a date range from these: dates = pl.date_range(df[0, "start"], df[0, "end"], "1mo", eager=True)  But I want to define a function which takes a couple of dates and spits out a range, as a wrapper around pl.date_range: def my_date_range(start: pl.Date, end: pl.Date) -> pl.Series: return pl.date_range(start, end, "1mo", eager=True)  The above doesn't typecheck with pyright/Pylance, because: Argument of type "Date" cannot be assigned to parameter "start" of type "IntoExprColumn | date | datetime" in function "date_range" Type "Date" is incompatible with type "IntoExprColumn | date | datetime" "Date" is incompatible with "date" "Date" is incompatible with "datetime" "Date" is incompatible with "Expr" "Date" is incompatible with "Series" "Date" is incompatible with "str"PylancereportArgumentType  If I check out type(df[0, "start"]), I see: datetime.date  and pl.Date is no good because isinstance(df[0, "start"], pl.Date) == False. I cannot figure out how to import datetime.date in order to use it as a type annotation (trying import polars.datetime as dt raises No module named 'polars.datetime'). How can this be done? Or put differently: how should my_date_range's date arguments be annotated?

Answer: Since datetime.date is compatible with the start and end parameters expected by pl.date_range() this should be sufficient: import polars as pl from datetime import date def my_date_range(start: date, end: date) -> pl.Series: return pl.date_range(start, end, "1mo", eager=True)
Question: I want to remove one field from a struct. Currently, I have it set up like this, but is there a simpler way to achieve this? import polars as pl import polars.selectors as cs def remove_one_field(df: pl.DataFrame) -> pl.DataFrame: meta_data_columns = (df.select('meta_data') .unnest('meta_data') .select(cs.all() - cs.by_name('system_data')).columns) print(meta_data_columns) return (df.unnest('meta_data') .select(cs.all() - cs.by_name('system_data')) .with_columns(meta_data=pl.struct(meta_data_columns)) .drop(meta_data_columns)) # Example usage input_df = pl.DataFrame({ "id": [1, 2], "meta_data": [{"system_data": "to_remove", "user_data": "keep"}, {"user_data": "keep_"}] }) output_df = remove_one_field(input_df) print(output_df)  ['user_data'] shape: (2, 2) ┌─────┬───────────┐ │ id ┆ meta_data │ │ --- ┆ --- │ │ i64 ┆ struct[1] │ ╞═════╪═══════════╡ │ 1 ┆ {"keep"} │ │ 2 ┆ {"keep_"} │ └─────┴───────────┘  Something like select on fields within a struct?

Answer: You can use struct.field() which can accept either list of strings or multiple string arguments. You know your DataFrame' schema() so you can easily create list of fields you want fields = [c[0] for c in input_df.schema["meta_data"] if c[0] != "system_data"] input_df.with_columns( meta_data = pl.struct( pl.col.meta_data.struct.field(fields) ) ) ┌─────┬───────────┐ │ id ┆ meta_data │ │ --- ┆ --- │ │ i64 ┆ struct[1] │ ╞═════╪═══════════╡ │ 1 ┆ {"keep"} │ │ 2 ┆ {"keep_"} │ └─────┴───────────┘
Question: My DataFrame: import pandas as pd df = pd.DataFrame( { 'a': list('xxxxxxxxxxyyyyyyyyy'), 'b': list('1111222333112233444') } )  Expected output is a list of groups:  a b 0 x 1 1 x 1 2 x 1 3 x 1 4 x 2 5 x 2 6 x 2 a b 4 x 2 5 x 2 6 x 2 7 x 3 8 x 3 9 x 3 a b 10 y 1 11 y 1 12 y 2 13 y 2 a b 12 y 2 13 y 2 14 y 3 15 y 3 a b 14 y 3 15 y 3 16 y 4 17 y 4 18 y 4  Logic: Grouping starts with df.groupby(['a', 'b']) and then after that I want to join each group with its previous one which gives me the expected output. Maybe the initial grouping that I mentioned is not necessary. Note that in the expected output a column cannot contain both x and y. Honestly overlapping rows is not what I have used to do when using groupby. So I don't know how to try to do it. I tried df.b.diff() but It is not even close.

Answer: You can combine groupby, itertools.pairwise and concat: from itertools import pairwise out = [pd.concat([a[1], b[1]]) for a, b in pairwise(df.groupby(['a', 'b']))]  Functional variant: from itertools import pairwise from operator import itemgetter out = list(map(pd.concat, pairwise(map(itemgetter(1), df.groupby(['a', 'b'])))))  Note that you might need to use sort=False in groupby if you want to keep the original order. Output: [ a b 0 x 1 1 x 1 2 x 1 3 x 1 4 x 2 5 x 2 6 x 2, a b 4 x 2 5 x 2 6 x 2 7 x 3 8 x 3 9 x 3, a b 7 x 3 8 x 3 9 x 3 10 y 1 11 y 1, a b 10 y 1 11 y 1 12 y 2 13 y 2, a b 12 y 2 13 y 2 14 y 3 15 y 3, a b 14 y 3 15 y 3 16 y 4 17 y 4 18 y 4]
Question: I'm trying to scrape a table from a webpage using Selenium and BeautifulSoup but I'm not sure how to get to the actual data using BeautifulSoup. webpage: https://leetify.com/app/match-details/5c438e85-c31c-443a-8257-5872d89e548c/details-general I tried extracting table rows (tag <tr>) but when I call find_all, the array is empty. When I inspect element, I see several elements with a tr tag, why don't they show up with BeautifulSoup.find_all() ?? I tried extracting table rows (tag <tr>) but when I call find_all, the array is empty. Code: from selenium import webdriver from bs4 import BeautifulSoup driver = webdriver.Chrome() driver.get("https://leetify.com/app/match-details/5c438e85-c31c-443a-8257-5872d89e548c/details-general") html_source = driver.page_source soup = BeautifulSoup(html_source, 'html.parser') table = soup.find_all("tbody") print(len(table)) for entry in table: print(entry) print("\n") 

Answer:  why don't they show up with BeautifulSoup.find_all() ??  after taking a quick glance, it seems like it takes a long time for the page to load. The thing is, when you pass the driver.page_source to BeautifulSoup, not all the HTML/CSS is loaded yet. So, the solution would be to use an Explicit wait: Wait until page is loaded with Selenium WebDriver for Python or, even, (less recommended): from time import sleep sleep(10)  but I'm not 100% sure, since I don't currently have Selenium installed on my machine  However, I'd like to take on a completely different solution: If you take a look at your browsers Network calls (Click on F12 in your browser, and it'll open the developer options), you'll see that data (the table) your looking for, is loaded through sending a GET request the their API:  The endpoint is under: https://api.leetify.com/api/games/5c438e85-c31c-443a-8257-5872d89e548c  which you can view directly from your browser. So, you can directly use the requests library to make a GET request to the above endpoint, which will be much more efficent: import requests from pprint import pprint response = requests.get('https://api.leetify.com/api/games/5c438e85-c31c-443a-8257-5872d89e548c') data = response.json() pprint(data)  Prints (trucated): {'agents': [{'gameFinishedAt': '2024-07-06T07:10:02.000Z', 'gameId': '5c438e85-c31c-443a-8257-5872d89e548c', 'id': '63e38340-d1ae-4e19-b51c-e278e3325bbb', 'model': 'customplayer_tm_balkan_variantk', 'steam64Id': '76561198062922849', 'teamNumber': 2}, {'gameFinishedAt': '2024-07-06T07:10:02.000Z', 'gameId': '5c438e85-c31c-443a-8257-5872d89e548c', 'id': 'e10f9fc4-759d-493b-a17f-a85db2fcd09d', 'model': 'customplayer_ctm_fbi_variantg', 'steam64Id': '76561198062922849', 'teamNumber': 3},  This approach bypasses the need to wait for the page to load, allowing you to directly access the data.
Question: I would like to add a legend to my Python plot, with a title and legend handles. My sincere apologies as a complete novice in Python, I got my code from a post. The code below works, but I want to add a legend. All the plots I have googled deal with line plots with several lines. import geopandas as gpd import matplotlib.pyplot as plt from datetime import date from mpl_toolkits.basemap import Basemap map_df = gpd.read_file("../Shapefiles/lso_adm_fao_mlgca_2019/lso_admbnda_adm1_FAO_MLGCA_2019.shx") risks_df=pd.read_csv("../Output/Wndrisks.csv") merged_df = map_df.merge(risks_df, left_on=["ADM1_EN"], right_on=["District"]) d = {1: "green", 2: "yellow", 3: "orange", 4: "red"} colors = map_df["ADM1_EN"].map(risks_df.set_index("District")["risk"].map(d)) ax = map_df.plot(color=colors, edgecolor="k", alpha=0.7, legend=True, legend_kwds={"label": "Risk Level", "orientation": "vertical"}) map = Basemap(projection='merc', llcrnrlon=26.5,llcrnrlat=-31.0,urcrnrlon=30.0,urcrnrlat=-28.5, epsg=4269) map.drawlsmask(land_color='grey',ocean_color='aqua',lakes=True) legend = plt.legend(handles=[one, two, three, four], title="Risk Levels", loc=4, fontsize='small', fancybox=True) plt.title(f"Strong Wind Risks 01-10Jun24", y=1.04) plt.tick_params( axis="both", # affect both the X and Y which="both", # get rid of both major and minor ticks top=False, # get rid of ticks top/bottom/left/right bottom=False, left=False, right=False, labeltop=False, # get rid of labels top/bottom/left/right labelbottom=False, labelleft=False, labelright=False) plt.axis("off") # Get rid of the border around the map plt.subplots_adjust(right=0.85) # Nudge the country to the left a bit plt.savefig('wndriskmap.png', dpi=300) plt.show()  The data is for the form: "District","risk" "Berea",3 "Butha-Buthe",4 "Leribe",4 "Mafeteng",4 "Maseru",4 "Mohale's Hoek",4 "Mokhotlong",4 "Qacha's Nek",4 "Quthing",4 "Thaba-Tseka",4  The plot I get is as attached I can attach the shapefile if required. I want the legend to have a title "Risk Level" and the levels 1=no risk, 2=low risk, 3=medium risk and 4=high risk. what I have included in legend = plt.legend(...) does not work. Assistance will be appreciated.

Answer: You could map the risks to the labels and make a categorical plot : from matplotlib.colors import ListedColormap colors = {1: "green", 2: "yellow", 3: "orange", 4: "red"} # or a list labels = {1: "no risk", 2: "low risk", 3: "medium risk", 4: "high risk"} catego = map_df["risk"].astype(str).str.cat(map_df["risk"].map(labels), sep="- ") fig, ax = plt.subplots(figsize=(5, 5)) map_df.plot( column=catego, categorical=True, edgecolor="k", alpha=0.7, cmap=ListedColormap(colors.values()), legend=True, legend_kwds={ "title": "Risk Level", "shadow": True, "loc": "lower right", }, ax=ax, ) ax.set_axis_off() ax.set_title("Strong Winds in 10-D Risks 01-10Jun")  NB: I used the risks as a prefix/workaround to preserve the labels' order in the legend.
Question: I'm accessing OpenStreetMap data using osmnx, using: import osmnx as ox graph = ox.graph_from_place('Bennekom') nodes, edges = ox.graph_to_gdfs(graph)  I know from openstreetmap.org/edit that all (?) street features have an attribute Surface, which can be Unpaved, Asphalt, Gravel, et cetera. However, that info is not included in the GeoDataFrames, so I cannot filter or select certain surface types. Is that somehow possible?

Answer: You need to include the surface key in the useful_tags_way : import osmnx as ox ox.settings.useful_tags_way = ["surface"] # << add this line graph = ox.graph_from_place("Bennekom") nodes, edges = ox.graph_to_gdfs(graph)  NB: You might need to explode the surface column in order to get all matches when filtering. print( edges.iloc[:, :-1] .sample(frac=1, random_state=111) .to_string(max_rows=10, max_cols=5, index=False) ) osmid surface oneway reversed length [1213295165, 6920271] [paving_stones, asphalt] False True 34.747 6920805 asphalt False True 69.793 189224820 asphalt False True 21.313 701436450 NaN False False 14.168 6920602 paving_stones False False 67.469 ... ... ... ... ... [1281296947, 604729524] asphalt False True 92.612 173209511 NaN False True 84.315 604729512 NaN False True 157.071 857498475 NaN False False 17.064 365986402 NaN False True 14.977
Question: I can't find a way to convert a binary to a string representation without using a UDF. Is there a way with native PySpark functions and not a UDF? from pyspark.sql import DataFrame, SparkSession import pyspark.sql.functions as F import uuid from pyspark.sql.types import Row, StringType spark_test_instance = (SparkSession .builder .master('local') .getOrCreate()) df: DataFrame = spark_test_instance.createDataFrame([Row()]) df = df.withColumn("id", F.lit(uuid.uuid4().bytes)) df = df.withColumn("length", F.length(df["id"])) uuidbytes_to_str = F.udf(lambda x: str(uuid.UUID(bytes=bytes(x), version=4)), StringType()) df = df.withColumn("id_str", uuidbytes_to_str(df["id"])) df = df.withColumn("length_str", F.length(df["id_str"])) df.printSchema() df.show(1, truncate=False)  gives: root |-- id: binary (nullable = false) |-- length: integer (nullable = false) |-- id_str: string (nullable = true) |-- length_str: integer (nullable = false) +-------------------------------------------------+------+------------------------------------+----------+ |id |length|id_str |length_str| +-------------------------------------------------+------+------------------------------------+----------+ |[0A 35 DC 67 13 C8 47 7E B0 80 9F AB 98 CA FA 89]|16 |0a35dc67-13c8-477e-b080-9fab98cafa89|36 | +-------------------------------------------------+------+------------------------------------+----------+ 

Answer: You can use hex for getting the id_str: from pyspark.sql import SparkSession import pyspark.sql.functions as F import uuid spark = SparkSession.builder.getOrCreate() data = [(uuid.uuid4().bytes,)] df = spark.createDataFrame(data, ["id"]) df = df.withColumn("id_str", F.lower(F.hex("id"))) df.show(truncate=False) # +-------------------------------------------------+--------------------------------+ # |id |id_str | # +-------------------------------------------------+--------------------------------+ # |[8C 76 42 18 BD CA 47 A1 9C D7 9D 74 0C 3C A4 76]|8c764218bdca47a19cd79d740c3ca476| # +-------------------------------------------------+--------------------------------+   Update: You can also use regexp_replace to get the exact id_str, in the same format you have shared, as follows: from pyspark.sql import SparkSession import pyspark.sql.functions as F import uuid spark = SparkSession.builder.getOrCreate() data = [(uuid.uuid4().bytes,)] df = spark.createDataFrame(data, ["id"]) df = df.withColumn( "id_str", F.regexp_replace( F.lower(F.hex("id")), "(.{8})(.{4})(.{4})(.{4})(.{12})", "$1-$2-$3-$4-$5" ) ) df.show(truncate=False) # +-------------------------------------------------+------------------------------------+ # |id |id_str | # +-------------------------------------------------+------------------------------------+ # |[F8 25 99 3E 2D 7A 40 E4 A1 24 C0 28 B9 30 F6 03]|f825993e-2d7a-40e4-a124-c028b930f603| # +-------------------------------------------------+------------------------------------+
Question: I'm creating a chatbot in VS Code where it will receive csv file through a prompt on Streamlit interface. However from the moment that file is loaded, it is showing a message with the following content:  ValueError: This agent relies on access to a python repl tool which can execute arbitrary code. This can be dangerous and requires a specially sandboxed environment to be safely used. Please read the security notice in the doc-string of this function. You must opt-in to use this functionality by setting allow_dangerous_code=True.For general security guidelines, please see: https://python.langchain.com/v0.2/docs/security/  Traceback File "c:\Users\ \langchain-ask-csv\.venv\Lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 589, in _run_script exec(code, module.__dict__) File "C:\Users\ \langchain-ask-csv\main.py", line 46, in <module> main() File "C:\Users\ \langchain-ask-csv\main.py", line 35, in main agent = create_csv_agent( OpenAI(), csv_file, verbose=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "c:\Users\ \langchain-ask-csv\.venv\Lib\site-packages\langchain_experimental\agents\agent_toolkits\csv\base.py", line 66, in create_csv_agent return create_pandas_dataframe_agent(llm, df, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "c:\Users\ T\langchain-ask-csv\.venv\Lib\site-packages\langchain_experimental\agents\agent_toolkits\pandas\base.py", line 248, in create_pandas_dataframe_agent raise ValueError(  Here's is part of the code where I'm passing the file: def main(): load_dotenv() # Load the OpenAI API key from the environment variable if os.getenv("OPENAI_API_KEY") is None or os.getenv("OPENAI_API_KEY") == "": print("OPENAI_API_KEY is not set") exit(1) else: print("OPENAI_API_KEY is set") st.set_page_config(page_title="Ask your CSV") st.header("Ask your CSV 📈") csv_file = st.file_uploader("Upload a CSV file", type="csv") if csv_file is not None: agent = create_csv_agent( OpenAI(), csv_file, verbose=True) user_question = st.text_input("Ask a question about your CSV: ") if user_question is not None and user_question != "": with st.spinner(text="In progress..."): st.write(agent.run(user_question)) if __name__ == "__main__": main()  I checked the link given as suggestion and also tried to search on similar reports but haven't had success. What might be wrong and how to fix it?

Answer: The referenced security notice is in https://api.python.langchain.com/en/latest/agents/langchain_experimental.agents.agent_toolkits.pandas.base.create_pandas_dataframe_agent.html. Just do what the message tells you. Do a security analysis, create a sandbox environment for your thing to run in, and then add allow_dangerous_code=True to the arguments you pass to create_csv_agent, which just forwards the argument to create_pandas_dataframe_agent and run it in the sandbox.
Question: Trying to build recursive types to annotate a nested data structure, I hit the following. This code is correct according to mypy: IntType = int | list["IntType"] | tuple["IntType", ...] StrType = str | list["StrType"] | tuple["StrType", ...] def int2str(x: IntType) -> StrType: if isinstance(x, list): return list(int2str(v) for v in x) if isinstance(x, tuple): return tuple(int2str(v) for v in x) return str(x)  But not this one, which should be equivalent: IntType = int | list["IntType"] | tuple["IntType", ...] StrType = str | list["StrType"] | tuple["StrType", ...] def bad_int2str(x: IntType) -> StrType: if isinstance(x, (list, tuple)): return type(x)(bad_int2str(v) for v in x) # error here return str(x)  The error message is line 6: error: Incompatible return value type ( got "list[int | list[IntType] | tuple[IntType, ...]] | tuple[int | list[IntType] | tuple[IntType, ...], ...]", expected "str | list[StrType] | tuple[StrType, ...]" ) [return-value] line 6: error: Generator has incompatible item type "str | list[StrType] | tuple[StrType, ...]"; expected "int | list[IntType] | tuple[IntType, ...]" [misc]  I would assume mypy could infer that type(x) is either list or tuple. Is this a limitation of mypy or is there something fishy with this code? If so, where does the limitation come from?

Answer: There's no type erasure for type(x). What should mypy say about the following? x: list[int] = [1] reveal_type(type(x))  If we ask, it says:  Revealed type is "type[builtins.list[builtins.int]]"  So, when you ask for type(x)(some_strtype_iterator), it rightfully complains that you try to construct a list[StrType] | tuple[StrType, ...] from an iterable of IntType. Both errors hint about that fact: mypy thinks that you return a list or tuple of IntType, because the error inside the expression does not make it lose an already known type. And it also points out that you can't build a list/tuple of IntType from a generator yielding StrType - you didn't intend to build IntType sequence, but type(x) mandates that. I just started a discussion on the typing forum to clarify the reasons for not erasing the generics for type(x).
Question: Rolling arithmetic mean can simply be computed with Numpy's 'convolve' function, but how could I efficiently create an array of running geometric means of some array a and a given window size? To give an example, for an array: [0.5 , 2.0, 4.0] and window size 2, (with window size decreasing at the edges) I want to quickly generate the array: [0.5, 1.0, 2.83, 4.0]

Answer: import numpy as np from numpy.lib.stride_tricks import sliding_window_view from scipy.stats import gmean window = 2 a = [0.5, 2.0, 4.0] padded = np.pad(a, window - 1, mode="constant", constant_values=np.nan) windowed = sliding_window_view(padded, window) result = gmean(windowed, axis=1, nan_policy="omit") print(result) # >>> [0.5 1. 2.82842712 4. ]   Use the built-in gmean() from scipy. Pad with nan, which, in combination with gmean(…, nan_policy="omit"), produces decreasing window sizes at the boundaries. Use sliding_window_view() to create the running result. Put everyting together (see above).  If you don't need the decreasing window sizes at the boundaries (this is referring to your comment), you can skip the padding step and choose the nan_policy that suits you best. Update: Realizing that gmean() provides a weights argument, we can replace the nan padding with an equivalent weights array (1 for the actual values, 0 for the padded values), and then are free again to choose the nan_policy of our liking, even in the case of decreasing window sizes at the boundaries. This means, we could write: padded = np.pad(a, window - 1, mode="constant", constant_values=1.) windowed = sliding_window_view(padded, window) weights = sliding_window_view( np.pad(np.ones_like(a), window - 1, mode="constant", constant_values=0.), window) result = gmean(windowed, axis=1, weights=weights)  – which will produce exactly the same result as above. My gut feeling tells me that the original version is faster, but I did not do any speed tests.
Question: I want to format a number so that the mantissa is between 0 and 1 instead of 1 and 10 in scientific notation. For example; a=7.365 print("{:.6e}".format(a))  This will output 7.365000e+00, but I want it to be 0.736500e+01.

Answer: I suppose you could always try constructing your own preformatted string. (No idea if this works in Python 2.7, though). import math def fixit( x, n ): if x == 0.0: return f" {x:.{n}e}" s = ' ' if x >= 0 else '-' y = math.log10( abs(x) ) m = math.floor(y) + 1 z = 10 ** ( y - m ) return s + f"{z:.{n}f}e{m:+03d}" for x in [ -7635, -763.5, -76.35, -7.635, -0.7635, -0.07635, -0.007635, 0.007635, 0.07635, 0.7635, 7.635, 76.35, 763.5, 7635 ]: print( fixit( x, 6 ) ) for x in [ -10, -1, -0.1, -0.01, 0.0, 0.01, 0.1, 1, 10 ]: print( fixit( x, 6 ) )  Output: -0.763500e+04 -0.763500e+03 -0.763500e+02 -0.763500e+01 -0.763500e+00 -0.763500e-01 -0.763500e-02 0.763500e-02 0.763500e-01 0.763500e+00 0.763500e+01 0.763500e+02 0.763500e+03 0.763500e+04 -0.100000e+02 -0.100000e+01 -0.100000e+00 -0.100000e-01 0.000000e+00 0.100000e-01 0.100000e+00 0.100000e+01 0.100000e+02