eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to find out if any of the value in columns mark1, mark2, mark3, mark4 and mark5 are the same, column-wise comparison from a dataframe below, and list result as True or False import pandas as pd df = pd.DataFrame(data=[[7, 2, 3, 7, 7], [3, 4, 3, 2, 7], [1, 6, 5, 2, 7], [5, 5, 6, 3, 1]], columns=["mark1", "mark2", 'mark3', 'mark4', 'mark5'])  Ideal output:  mark1 mark2 mark3 mark4 mark5 result 0 7 2 3 7 7 True 1 3 4 3 2 7 True 2 1 6 5 2 7 False 3 5 5 6 3 1 True  So I came up with a func using nested forloop to compare each value in a column, does not work. AttributeError: 'Series' object has no attribute 'columns' What's the correct way? Avoid nested forloop by all means. def compare_col(df): check = 0 for i in range(len(df.columns.tolist())+1): for j in range(1, len(df.columns.tolist())+1): if df.iloc[i, i] == df.iloc[j, i]: check += 1 if check >= 1: return True else: return False df['result'] = df.apply(lambda x: compare_col(x[['mark1', 'mark2', 'mark3', 'mark4', 'mark5]]), axis=1) 

Answer: No need to use apply or a loop, compare the output of nunique to the number of columns: df['result'] = df.nunique(axis=1).ne(df.shape[1])  Output:  mark1 mark2 mark3 mark4 mark5 result 0 7 2 3 7 7 True 1 3 4 3 2 7 True 2 1 6 5 2 7 False 3 5 5 6 3 1 True  If you want a more efficient method and assuming a reasonable number of columns (less than a thousand) and numbers, you could use numpy to sort the values, compute the diff and check whether any value is 0: import numpy as np df['result'] = (np.diff(np.sort(df), axis=1)==0).any(axis=1)  Output:  mark1 mark2 mark3 mark4 mark5 result 0 7 2 3 7 7 True 1 3 4 3 2 7 True 2 1 6 5 2 7 False 3 5 5 6 3 1 True
Question: I write the main application with python and build it to windows *.exe file, and now need to call a 3rd party perl script. I use, cmd = "perl fileconverter.pl" subprocess.call(cmd)  Everything works well except the cmd window, which will pop up when subprocess.call(cmd) is called. Question is, how to hide the cmd window in this situation?

Answer: You can use wperl instead of perl.  If your program is running in a console, perl should reuse the same console. >perl -e"system 'perl -Esay+123'" 123  (I don't have Python installed, but the same applies no matter what launches perl.) But maybe the Python is running without a console. Then perl will create itself a console. Another answer shows you how to hide this console when using Python, and it's a great solution. But there's a simple Perl-specific option you can also use. Strawberry Perl (and I think ActivePerl, and probably all builds of Perl for Windows) provide not just perl.exe, but wperl.exe.[1] The latter is identical to the former, except it uses a hidden console.[2] So all you would need to do is change perl to wperl.   I don't count builds of Perl for unix emulation environments (Cygwin, MSYS, MSYS2) as Windows builds of Perl. The binaries differ by only one bit.
Question: I'm following a course on full-stack with Flask. My init.py looks like: from flask import Flask from config import Config from flask_mongoengine import MongoEngine app = Flask(__name__) app.config.from_object(Config) db = MongoEngine() db.init_app(app) from application import routes  However, when importing from flask_mongoengine import MongoEngine, I'm getting an ImportError: ImportError: cannot import name 'JSONEncoder' from 'flask.json'  My venv looks like: blinker==1.6.2 click==8.1.3 colorama==0.4.6 dnspython==2.3.0 email-validator==2.0.0.post2 Flask==2.3.2 flask-mongoengine==1.0.0 Flask-WTF==1.1.1 idna==3.4 itsdangerous==2.1.2 Jinja2==3.1.2 MarkupSafe==2.1.3 mongoengine==0.27.0 pymongo==4.4.0 python-dotenv==1.0.0 Werkzeug==2.3.6 WTForms==3.0.1  Is there anything I can do here to avoid this conflict? Thanks!

Answer: flask_mongoengine seems to be not currently maintained and does not work with current Flask versions. If you absolutely must use it, you need to downgrade your Flask version, which may (and likely will) get you into other trouble. There is an issue on github regarding your error message: https://github.com/MongoEngine/flask-mongoengine/issues/522 The deprecation warning came with Flask 2.2.0 in 08/2022: Flask Changes After a brief look at the repo, it seems the maintainer was already on it: https://github.com/MongoEngine/flask-mongoengine/blob/master/flask_mongoengine/json.py
Question: I have the following pandas dataframe:  col 2023-05-04 10:34:51.002100665 True 2023-05-04 10:34:51.007100513 True 2023-05-04 10:34:51.012100235 True 2023-05-04 10:34:51.017100083 False 2023-05-04 10:34:51.022099789 False 2023-05-04 10:35:23.610740595 False 2023-05-04 10:35:23.615740466 True 2023-05-04 10:35:23.620740227 True 2023-05-04 10:35:23.625740082 False 2023-05-04 10:35:23.630739797 True  How do I count the number of True and False blocks present in it? The result should be: Number of True blocks: 3 Number of False blocks: 2 

Answer: Assuming 'col' your column, you can remove the identical successive values with boolean indexing and use value_counts: out = df.loc[df['col'].ne(df['col'].shift()), 'col'].value_counts()  Output: col True 3 False 2 dtype: int64  Intermediates: * only those rows will be considered  col shift ≠ shift 2023-05-04 10:34:51.002100665 True NaN True * 2023-05-04 10:34:51.007100513 True True False 2023-05-04 10:34:51.012100235 True True False 2023-05-04 10:34:51.017100083 False True True * 2023-05-04 10:34:51.022099789 False False False 2023-05-04 10:35:23.610740595 False False False 2023-05-04 10:35:23.615740466 True False True * 2023-05-04 10:35:23.620740227 True True False 2023-05-04 10:35:23.625740082 False True True * 2023-05-04 10:35:23.630739797 True False True *
Question: In pandas, I can interpolate based on a datetimes like this: df1 = pd.DataFrame( { "ts": [ datetime(2020, 1, 1), datetime(2020, 1, 3, 0, 0, 12), datetime(2020, 1, 3, 0, 1, 35), datetime(2020, 1, 4), ], "value": [1, np.nan, np.nan, 3], } ) df1.set_index('ts').interpolate(method='index')  Outputs:  value ts 2020-01-01 00:00:00 1.000000 2020-01-03 00:00:12 2.333426 2020-01-03 00:01:35 2.334066 2020-01-04 00:00:00 3.000000  Is there a similar method in polars? Say, starting with df1 = pl.DataFrame( { "ts": [ datetime(2020, 1, 1), datetime(2020, 1, 3, 0, 0, 12), datetime(2020, 1, 3, 0, 1, 35), datetime(2020, 1, 4), ], "value": [1, None, None, 3], } )  shape: (4, 2) ┌─────────────────────┬───────┐ │ ts ┆ value │ │ --- ┆ --- │ │ datetime[μs] ┆ i64 │ ╞═════════════════════╪═══════╡ │ 2020-01-01 00:00:00 ┆ 1 │ │ 2020-01-03 00:00:12 ┆ null │ │ 2020-01-03 00:01:35 ┆ null │ │ 2020-01-04 00:00:00 ┆ 3 │ └─────────────────────┴───────┘  EDIT: I've updated the example to make it a bit more "irregular", so that upsample can't be used as a solution and to make it clear that we need something more generic

Answer: Update: Expr.interpolate_by was added in Polars 0.20.28 df1.with_columns(pl.col("value").interpolate_by("ts"))  shape: (4, 2) ┌─────────────────────┬──────────┐ │ ts ┆ value │ │ --- ┆ --- │ │ datetime[μs] ┆ f64 │ ╞═════════════════════╪══════════╡ │ 2020-01-01 00:00:00 ┆ 1.0 │ │ 2020-01-03 00:00:12 ┆ 2.333426 │ │ 2020-01-03 00:01:35 ┆ 2.334066 │ │ 2020-01-04 00:00:00 ┆ 3.0 │ └─────────────────────┴──────────┘
Question: I have found very easy and useful to load world map from geopandas datasets, as probably many others, for example: import geopandas as gpd world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres')) However, this gives a FutureWarning that dataset module is deprecated and will be removed in the future. There are maps available for download, for example from https://www.naturalearthdata.com/downloads/110m-cultural-vectors/ but the files are zipped and it does not seem like a convinient workflow to either get and process files from there or neither include processed files with the source. Is there an alternative? What is the best way to do this, especially if I want my code to work with future versions of Geopandas?

Answer: You can read it from Nacis : import geopandas as gpd url = "https://naciscdn.org/naturalearth/110m/cultural/ne_110m_admin_0_countries.zip" gdf = gpd.read_file(url)   old answer: The simplest solution would be to download/store the shapefile somewhere. That being said, if (for some reason), you need to read it from the source, you can do it this way : import fsspec url = "https://www.naturalearthdata.com/http//www.naturalearthdata.com/" \ "download/110m/cultural/ne_110m_admin_0_countries.zip" with fsspec.open(f"simplecache::{url}") as file: gdf = gpd.read_file(file)  Output :  featurecla scalerank ... FCLASS_UA geometry 0 Admin-0 country 1 ... None MULTIPOLYGON (((180.00000 -16.0... 1 Admin-0 country 1 ... None POLYGON ((33.90371 -0.95000, 34... 2 Admin-0 country 1 ... None POLYGON ((-8.66559 27.65643, -8... .. ... ... ... ... ... 174 Admin-0 country 1 ... Unrecognized POLYGON ((20.59025 41.85541, 20... 175 Admin-0 country 1 ... None POLYGON ((-61.68000 10.76000, -... 176 Admin-0 country 1 ... None POLYGON ((30.83385 3.50917, 29.... [177 rows x 169 columns]
Question: I am using polars for all preprocessing and feature engineering. I want to shuffle the data before performing a train/valid/test split. A training 'example' consists of multiple rows. The number of rows per example varies. Here is a simple contrived example (Note I am actually using a LazyFrame in my code): pl.DataFrame({ "example_id": [1, 1, 2, 2, 2, 3, 3, 3, 4, 4], "other_col": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] })  ┌────────────┬───────────┐ │ example_id ┆ other_col │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞════════════╪═══════════╡ │ 1 ┆ 1 │ │ 1 ┆ 2 │ │ 2 ┆ 3 │ │ 2 ┆ 4 │ │ 2 ┆ 5 │ │ 3 ┆ 6 │ │ 3 ┆ 7 │ │ 3 ┆ 8 │ │ 4 ┆ 9 │ │ 4 ┆ 10 │ └────────────┴───────────┘  I want to shuffle 'over' the example_id column, while keeping the examples grouped together. Producing a result something like this: ┌────────────┬───────────┐ │ example_id ┆ other_col │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞════════════╪═══════════╡ │ 2 ┆ 3 │ │ 2 ┆ 4 │ │ 2 ┆ 5 │ │ 1 ┆ 1 │ │ 1 ┆ 2 │ │ 4 ┆ 9 │ │ 4 ┆ 10 │ │ 3 ┆ 6 │ │ 3 ┆ 7 │ │ 3 ┆ 8 │ └────────────┴───────────┘  I then want to split the data fractionally, for example 0.6, 0.2, 0.2 for training, validation and testing respectively, but do this based on 'whole examples' rather than just row wise. Is there a clean way to do this in polars without having to convert the example_id to an array, shuffling it, splitting into sublists, then reselecting from the original dataframe?

Answer: There must be a far cleaner way of achieving this, hopefully someone can improve on this. Also it requires collecting the dataframe which is not ideal. Either way, it seems to work for now. Thanks @jqurious for the pointer.  Grab the unique example_ids, shuffle them and add a row count.  example_ids = ( example_df .select("example_id") .unique() .sample(fraction=1, shuffle=True) .with_row_index() )   Split the unique ids into subsets using the row count.  # assume we'll test on remaining data train_frac = 0.6 valid_frac = 0.2 train_ids = example_ids.filter( pl.col("index") < pl.col("index").max() * train_frac ) valid_ids = example_ids.filter( pl.col("index").is_between( pl.col("index").max() * train_frac, pl.col("index").max() * (train_frac + valid_frac), ) ) test_ids = example_ids.filter( pl.col("index") > pl.col("index").max() * (train_frac + valid_frac) )   Join each subset back to the example_df and drop the row_nr  train_df = example_df.join(train_ids, on="example_id").drop("index") valid_df = example_df.join(valid_ids, on="example_id").drop("index") test_df = example_df.join(test_ids, on="example_id").drop("index")  This will produce 3 dataframe, something like this ┌────────────┬───────────┐ │ example_id ┆ other_col │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞════════════╪═══════════╡ │ 1 ┆ 1 │ │ 1 ┆ 2 │ │ 3 ┆ 6 │ │ 3 ┆ 7 │ │ 3 ┆ 8 │ └────────────┴───────────┘  ┌────────────┬───────────┐ │ example_id ┆ other_col │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞════════════╪═══════════╡ │ 2 ┆ 3 │ │ 2 ┆ 4 │ │ 2 ┆ 5 │ └────────────┴───────────┘  ┌────────────┬───────────┐ │ example_id ┆ other_col │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞════════════╪═══════════╡ │ 4 ┆ 9 │ │ 4 ┆ 10 │ └────────────┴───────────┘
Question: When evaluating list elements I would like to know and use the current index. Is there already a way of doing it? Something like pl.element().idx() ? import polars as pl data = {"a": [[1,2,3],[4,5,6]]} schema = {"a": pl.List(pl.Int8)} df = pl.DataFrame(data, schema=schema) df = df.with_columns( pl.col("a").list.eval(pl.element() * pl.element().idx()) ) # AttributeError: 'Expr' object has no attribute 'idx'  Expected result: +-------------+ ¦ a ¦ ¦ --- ¦ ¦ list[u8] ¦ ¦-------------¦ ¦ [0, 2, 6] ¦ ¦ [0, 5, 12] ¦ +-------------+ 

Answer: The best way (that I know of) is to make a row index, explode, use int_range with a window function to create the idx (I'm calling it j), and then put it back together with group_by/agg ( df .with_row_index('i') .explode('a') .with_columns(j=pl.int_range(pl.len()).over('i')) .with_columns(new=pl.col('a')*pl.col('j')) .group_by('i', maintain_order=True) .agg(pl.col('new')) .drop('i') )
Question: I have a code df.select([ pl.all().exclude("elapsed_time_linreg"), pl.col("elapsed_time_linreg").arr.get(0).suffix("_slope"), pl.col("elapsed_time_linreg").arr.get(1).suffix("_intercept"), pl.col("elapsed_time_linreg").arr.get(2).suffix("_resid_std"), ])  which unpacked the result of a function @jit def linear_regression(session_np: np.ndarray) -> np.ndarray: w = len(session_np) x = np.arange(w) sx = w ** 2 / 2 sy = np.sum(session_np) sx2 = (w * (w + 1) * (2 * w + 1)) / 6 sxy = np.sum(x * session_np) slope = (w * sxy - sx * sy) / (w * sx2 - sx**2) intercept = (sy - slope * sx) / w resids = session_np - (x * slope + intercept) return slope, intercept, resids.std() def get_linreg_aggs(session) -> np.ndarray: return linear_regression(np.array(session)) df.select( pl.col("elapsed_time") .apply(utils.get_linreg_aggs) .cast(pl.Float32) .alias("elapsed_time_linreg") )  which worked perfectly on polars 17.15, but since I upgraded it to the latest (0.18.4), it stopped working. Now I get the following error and have no idea how to fix this: AttributeError: 'ExprArrayNameSpace' object has no attribute 'get'  Is there a way of fixing it without downgrading the version? Or even a more efficient way of creating a bunch of columns from this function? I thought of the caching the computational result with lru-cache, but since polars uses multiprocessing, idk if it helps.

Answer: The .arr namespace was renamed to .list in v0.18.0 The function is now .list.get()
Question: I'm trying to quote one of the values I send to an f-string in Python: f'This is the value I want quoted: \'{value}\''  This works, but I wonder if there's a formatting option that does this for me, similar to how %q works in Go. Basically, I'm looking for something like this: f'This is the value I want quoted: {value:q}' >>> This is the value I want quoted: 'value'  I would also be okay with double-quotes. Is this possible?

Answer: Use the explicit conversion flag !r: >>> value = 'foo' >>> f'This is the value I want quoted: {value!r}' "This is the value I want quoted: 'foo'"  The r stands for repr; the result of f'{value!r}' should be equivalent to using f'{repr(value)}'. Conversion flags are a feature carried over from str.format, which predates f-strings. For some reason undocumented in PEP 3101, but mentioned in the docs, there's also an !a flag which converts with ascii: >>> f'quote {"🔥"!a}' "quote '\\U0001f525'"  And there's an !s for str, which seems useless... unless you know that objects can override their formatter to do something different than object.__format__ does. It provides a way to opt-out of those shenanigans and use __str__ anyway. >>> class What: ... def __format__(self, spec): ... if spec == "fancy": ... return "𝓅𝑜𝓉𝒶𝓉𝑜" ... return "potato" ... def __str__(self): ... return "spam" ... def __repr__(self): ... return "<wacky object at 0xcafef00d>" ... >>> obj = What() >>> f'{obj}' 'potato' >>> f'{obj:fancy}' '𝓅𝑜𝓉𝒶𝓉𝑜' >>> f'{obj!s}' 'spam' >>> f'{obj!r}' '<wacky object at 0xcafef00d>'  One example of where that !s might be useful in practice is with alignment/padding of types that don't otherwise support it: >>> f"{[1,2]:.>10}" TypeError: unsupported format string passed to list.__format__ >>> f"{[1,2]!s:.>10}" '....[1, 2]'  Or types which "support" it, but not how you may have intended: >>> from datetime import date >>> f"{date.today():_^14}" '_^14' >>> f"{date.today()!s:_^14}" '__2024-03-25__'
Question: I've searched all over langchain documentation on their official website but I didn't find how to create a langchain doc from a str variable in python so I searched in their GitHub code and I found this :  doc=Document( page_content="text", metadata={"source": "local"} )  PS: I added the metadata attribute then I tried using that doc with my chain: Memory and Chain: memory = ConversationBufferMemory(memory_key="chat_history", input_key="human_input") chain = load_qa_chain( llm, chain_type="stuff", memory=memory, prompt=prompt )  the call method:  chain({"input_documents": doc, "human_input": query})  prompt template: template = """You are a senior financial analyst analyzing the below document and having a conversation with a human. {context} {chat_history} Human: {human_input} senior financial analyst:""" prompt = PromptTemplate( input_variables=["chat_history", "human_input", "context"], template=template )  but I am getting the following error: AttributeError: 'tuple' object has no attribute 'page_content'  when I tried to check the type and the page content of the Document object before using it with the chain I got this print(type(doc)) <class 'langchain.schema.Document'> print(doc.page_content) "text" 

Answer: I had a similar issue and I noticed the API calls for a list, so try  doc = Document(page_content=input, metatdata={ "source": "userinput" } ) #db.add_documents(doc) db.add_documents([doc])
Question: I am trying to create a test for an API I wrote using Django-Ninja. Here is my Model: class Country(models.Model): created_at = models.DateTimeField(auto_created=True, auto_now_add=True) name = models.CharField(max_length=128, null=False, blank=False) code = models.CharField(max_length=128, null=False, blank=False, unique=True) timezone = models.CharField(max_length=128, null=False, blank=False)  Here is my schema: class CountryAddSchema(Schema): name: str code: str timezone: str  Here is the post endpoint: router.post("/add", description="Add a Country", summary="Add a Country", tags=["Address"], response={201: DefaultSchema, 401: DefaultSchema, 422: DefaultSchema, 500: DefaultSchema}, url_name="address_country_add") def country_add(request, country: CountryAddSchema): try: if not request.auth.belongs_to.is_staff: return 401, {"detail": "None Staff cannot add Country"} the_country = Country.objects.create(**country.dict()) the_country.save() return 201, {"detail": "New Country created"} except Exception as e: return 500, {"detail": str(e)}  Finally, here the test function: def test_add_correct(self): """ Add a country """ data = { "name": "".join(choices(ascii_letters, k=32)), "code": "".join(choices(ascii_letters, k=32)), "timezone": "".join(choices(ascii_letters, k=32)) } respond = self.client.post(reverse("api-1.0.0:address_country_add"), data, **self.AUTHORIZED_HEADER) self.assertEquals(respond.status_code, 201) self.assertDictEqual(json.loads(respond.content), {"detail": "New Country created"}) the_country = Country.objects.last() self.assertDictEqual( data, { "name": the_country.name, "code": the_country.code, "timezone": the_country.timezone } )  Please notice I have self.AUTHORIZED_HEADER set in setUp. And here the error: FAIL: test_add_correct (address.tests_country.CountryTest) Add a country ---------------------------------------------------------------------- Traceback (most recent call last): File "SOME_PATH/tests_country.py", line 80, in test_add_correct self.assertEquals(respond.status_code, 201) AssertionError: 400 != 201  I can add a country using swagger provided with django-ninja. I mean the endpoint works. But I can not test it using djano.test.Client. Any Idea? Update: Here the curl code generated by swagger: curl -X 'POST' \ 'http://127.0.0.1:8000/api/address/country/add' \ -H 'accept: application/json' \ -H 'X-API-Key: API-KEY' \ -H 'Content-Type: application/json' \ -d '{ "name": "string", "code": "string", "timezone": "string" }' 

Answer: I had {"detail": "Cannot parse request body"} as an error. Turns out, Django-ninja expects your data to be passed as a json, but by default, the content_type is set to multipart/form-data; boundary=BoUnDaRyStRiNg for the test client. When you explicitely mention the content type should be json, it will work. Client().post(url, {"your": "dict"}, content_type="application/json")  Note: Setting the content-type in the headers will not work as it will get overwritten.
Question: If I define a Protocol and a class that implements that protocol, I would like to document that the class does/should implement the protocol so that other developers can easily see that the class is supposed to implement the protocol, and so that static analysis tools can verify that the class implements the protocol. What is the proper way to document that in the class? Is it okay to subclass the protocol? Consider the protocol: from typing import Protocol class MyProtocol(Protocol): def foo() -> None: ...  Is it correct to define a class like this?: from my_protocol import MyProtocol class MyClass(MyProtocol): def __init__(self): pass def foo() -> None: pass  I could use an abstract base class, but then I wouldn't be able to support classes that are structural sub types, but not actual derived classes. I saw it looked like a version of this question was asked here, but it didn't seem to have an actual answer: Correct way to hint that a class is implementing a Protocol?

Answer: I think the answer is on PEP-0544 :  "To explicitly declare that a certain class implements a given protocol, it can be used as a regular base class."  https://peps.python.org/pep-0544/#explicitly-declaring-implementation Personally I do subclass from protocol often because:  my IDE is able to automatically complete method signature when I type them (the one defined inside the Protocol) my IDE can rename all method classes if I rename them from the Protocol I can implement deprecated methods (which call another method) in the Protocol class and not into all subclasses:  class DataGetter(Protocol): def get_data(self): ... def get(self): warn('get method is deprecated use get_data instead', DeprecationWarning) return self.get_data() class Image(DataGetter): def get_data(self): return "some image data"