eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have two dataframes that I am merging: import pandas as pd df = pd.DataFrame({'ID': [1,2,3,4], 'Column A': [2, 3, 2, 3], 'Column B': [None, None, 3, None]}) df2 = pd.DataFrame({'ID': [1,2,4], 'Column C': [2, 3, 22]}) df = df.merge(df2, on='ID', how='left') df  The output is:  ID Column A Column B Column C 0 1 2 NaN 2.0 1 2 3 NaN 3.0 2 3 2 3.0 NaN 3 4 3 NaN 22.0  Position 2 on the new 'Column C' is Null because ID=3 does not exist in df2. How can I fill the nulls in the new columns only but leave the nulls in the old column intact? Expected output:  ID Column A Column B Column C 0 1 2 NaN 2.0 1 2 3 NaN 3.0 2 3 2 3.0 0.0 3 4 3 NaN 22.0 

Answer: You can use the IDs to generate a mask for boolean indexing: out = df.merge(df2, on='ID', how='left') m = ~df['ID'].isin(df2['ID']) out[m] = out[m].fillna(0)  Alternatively, reindex and concat: out = pd.concat([df.set_index('ID'), df2.set_index('ID') .reindex(df['ID'], fill_value=0)], axis=1)  Another (not as nice) option, use a placeholder (e.g. object) for existing NaNs: out = (df.fillna(object) .merge(df2.fillna(object), on='ID', how='left') .fillna(0).replace({object: float('nan')}) )  Output:  ID Column A Column B Column C 0 1 2 NaN 2.0 1 2 3 NaN 3.0 2 3 2 3.0 0.0 3 4 3 NaN 22.0
Question: In Python 3.7 I read an png image with opencv and convert it to an jpg image stream. How can I determine the number of bytes of the stream? image=cv2.imread('test.png',0) image_stream = io.BytesIO(cv2.imencode(".jpg",image)[1].tobytes()) 

Answer: (success, data) = cv2.imencode(".jpg", image) assert success print(data.nbytes)  This number of bytes is identical to the size of the bytes object returned by the .tobytes() call. Creating a BytesIO() also does not affect this size.
Question: I have a pandas dataframe where the column names are capital and snake case. I want to convert them into camel case with first world starting letter to be lower case. The following code is not working for me. Please let me know how to fix this. import pandas as pd # Sample DataFrame with column names data = {'RID': [1, 2, 3], 'RUN_DATE': ['2023-01-01', '2023-01-02', '2023-01-03'], 'PRED_VOLUME_NEXT_360': [100, 150, 200]} df = pd.DataFrame(data) # Convert column names to lowercase df.columns = df.columns.str.lower() # Convert column names to camel case with lowercase starting letter df.columns = [col.replace('_', ' ').title().replace(' ', '').replace(col[0], col[0].lower(), 1) for col in df.columns] # Print the DataFrame with updated column names print(df)  I want to column names RID, RUN_DATE, PRED_VOLUME_NEXT_360 to be converted to rid, runDate, predVolumeNext360, but the code is giving Rid, RunDate and PredVolumeNext360.

Answer: You could use a regex to replace _x by _X: df.columns = (df.columns.str.lower() .str.replace('_(.)', lambda x: x.group(1).upper(), regex=True) )  Or with a custom function: def to_camel(s): l = s.lower().split('_') l[1:] = [x.capitalize() for x in l[1:]] return ''.join(l) df = df.rename(columns=to_camel)  Output:  rid runDate predVolumeNext360 0 1 2023-01-01 100 1 2 2023-01-02 150 2 3 2023-01-03 200
Question: I have been reading this other post, since I am dealing with a similar situation. However, I have a problem. In my version of df_1, I have timestamps which are outside of the values of the time ranges presented in df_2. Let's say I have an extra row print df_1 timestamp A B 0 2016-05-14 10:54:33 0.020228 0.026572 1 2016-05-14 10:54:34 0.057780 0.175499 2 2016-05-14 10:54:35 0.098808 0.620986 3 2016-05-14 10:54:36 0.158789 1.014819 4 2016-05-14 10:54:39 0.038129 2.384590 5 2023-11-22 10:54:39 0.000500 6.258710 print df_2 start end event 0 2016-05-14 10:54:31 2016-05-14 10:54:33 E1 1 2016-05-14 10:54:34 2016-05-14 10:54:37 E2 2 2016-05-14 10:54:38 2016-05-14 10:54:42 E3  I need to know how can I modify the solution to the previous post df_2.index = pd.IntervalIndex.from_arrays(df_2['start'],df_2['end'],closed='both') df_1['event'] = df_1['timestamp'].apply(lambda x : df_2.iloc[df_2.index.get_loc(x)]['event'])  so that I get a null value for the fifth row, since I am getting now an error

Answer: I would use janitor's conditional_join that enables a left join easily and much more efficiently than using apply: import janitor out = df_1.conditional_join(df_2, ('timestamp', 'start', '>='), ('timestamp', 'end', '<='), right_columns=['event'], how='left')  Or, if your intervals are non-overlapping and you expect a single match, using merge_asof and a mask: tmp = pd.merge_asof(df_1[['timestamp']], df_2.sort_values(by=['start', 'end']), left_on='timestamp', right_on='start') df_1['event'] = df_1['event'] = tmp['event'].where(tmp['timestamp'] .between(tmp['start'], tmp['end'], inclusive='both'))  Output:  timestamp A B event 0 2016-05-14 10:54:33 0.020228 0.026572 E1 1 2016-05-14 10:54:34 0.057780 0.175499 E2 2 2016-05-14 10:54:35 0.098808 0.620986 E2 3 2016-05-14 10:54:36 0.158789 1.014819 E2 4 2016-05-14 10:54:39 0.038129 2.384590 E3 5 2023-11-22 10:54:39 0.000500 6.258710 NaN
Question: I write a script to find out the fastest one in a list of cdn hosts: #!/usr/bin/env python3.11 import time from contextlib import contextmanager from enum import StrEnum import anyio import httpx @contextmanager def timeit(msg: str): start = time.time() yield cost = time.time() - start print(msg, f"{cost = }") class CdnHost(StrEnum): jsdelivr = "https://cdn.jsdelivr.net/npm/swagger-ui-dist@5.9.0/swagger-ui.css" unpkg = "https://unpkg.com/swagger-ui-dist@5.9.0/swagger-ui.css" cloudflare = ( "https://cdnjs.cloudflare.com/ajax/libs/swagger-ui/5.9.0/swagger-ui.css" ) TIMEOUT = 5 LOOP_INTERVAL = 0.1 async def fetch(client, url, results, index): try: r = await client.get(url) except (httpx.ConnectError, httpx.ReadError): ... else: print(f"{url = }\n{r.elapsed = }") if r.status_code < 300: results[index] = r.content class StopNow(Exception): ... async def find_fastest_host(timeout=TIMEOUT, loop_interval=LOOP_INTERVAL) -> str: urls = list(CdnHost) results = [None] * len(urls) try: async with anyio.create_task_group() as tg: with anyio.move_on_after(timeout): async with httpx.AsyncClient() as client: for i, url in enumerate(urls): tg.start_soon(fetch, client, url, results, i) for _ in range(int(timeout / loop_interval) + 1): for res in results: if res is not None: raise StopNow await anyio.sleep(0.1) except ( StopNow, httpx.ReadError, httpx.ReadTimeout, httpx.ConnectError, httpx.ConnectTimeout, ): ... for url, res in zip(urls, results): if res is not None: return url return urls[0] async def main(): with timeit("Sniff hosts"): url = await find_fastest_host() print("cdn host:", CdnHost) print("result:", url) if __name__ == "__main__": anyio.run(main)  There are three cdn hosts (https://cdn.jsdelivr.net, https://unpkg.com, https://cdnjs.cloudflare.com). I make three concurrent async task to fetch them by httpx. If one of them get a response with status_code<300, then stop all task and return the right url. But I don't know how to cancel tasks without using a custom exception (in the script is StopNow).

Answer: You can call the cancel method of the cancel_scope attribute of the task group to cancel all of its tasks: async with anyio.create_task_group() as tg: ... tg.cancel_scope.cancel()
Question: So i a .csv that has a simplified machine programm that looks like this: X51.972,Y1433.401 LASER_ON X41.972 Y1438.401 X51.97 LASER_OFF X51.972,Y1382.401 LASER_ON X41.972 Y1377.401 X51.97  When there is only an X value, I want to write the Y value from the line above next to the X value and when there is only a Y value i want it to write the X value next to it. lines: list[tuple[str, str]] = [] with open(output_csv_file) as input_file: for line in map(str.strip, input_file): if line: a, *b = line.split(",") if a[0] == "X": if b: lines.append((a, b[0])) else: lines.append((a, lines[-1][1])) else: assert a[0] == "Y" if b: lines.append((b[0], a)) else: lines.append((lines[-1][0], a)) with open(output_csv_file, 'w', newline ='') as output_file: csv_writer = csv.writer(output_file) for line in lines: csv_writer.writerow(line)  I managed to do what i want with this script but this only works, when there are no lines, that contain "LASER_ON" and LASER_OFF". Can i modify this to skip the lines containing these words?

Answer: The key is that you have to track the last values of X and Y. If there's a LASER line, you leave those two values alone but save the record. Also, you can't really use the CSV module to write the data if the lines are variable in length. output_csv_file = 'x.csv' lines = [] X = 'X0' Y = 'Y0' with open(output_csv_file) as input_file: for line in map(str.strip, input_file): if not line: continue if line.startswith('LASER'): lines.append([line]) else: row = line.split(",") if len(row) == 2: X,Y = row elif row[0][0] == 'X': X = row[0] else: Y = row[0] lines.append((X,Y)) with open(output_csv_file, 'w') as output_file: for line in lines: print( ','.join(line), file=output_file )  Output: X51.972,Y1433.401 LASER_ON X41.972,Y1433.401 X41.972,Y1438.401 X51.97,Y1438.401 LASER_OFF X51.972,Y1382.401 LASER_ON X41.972,Y1382.401 X41.972,Y1377.401 X51.97,Y1377.401
Question: I'm trying to build a simple RAG, and I'm stuck at this code: from langchain.embeddings.huggingface import HuggingFaceEmbeddings from llama_index import LangchainEmbedding, ServiceContext embed_model = LangchainEmbedding( HuggingFaceEmbeddings(model_name="thenlper/gte-large") ) service_context = ServiceContext.from_defaults( chunk_size=256, llm=llm, embed_model=embed_model ) index = VectorStoreIndex.from_documents(documents, service_context=service_context)  where I get ImportError: cannot import name 'LangchainEmbedding' from 'llama_index' How can I solve? Is it related to the fact that I'm working on Colab?

Answer: INFO 2024: This answer worked in 2023 when question was asked but it seems they moved all code and now you have to use other answers.  Not from llama_index import LangchainEmbedding  but from llama_index.embeddings import LangchainEmbedding  (See source code for llama_index/embeddings/__ init__.py)
Question: I'm trying to find the equivalent of a min_count param on polars groupby, such as in pandas.groupby(key).sum(min_count=N). Let's suppose the dataframe df = pl.from_repr(""" ┌───────┬───────┐ │ fruit ┆ price │ │ --- ┆ --- │ │ str ┆ i64 │ ╞═══════╪═══════╡ │ a ┆ 1 │ │ a ┆ 3 │ │ a ┆ 5 │ │ b ┆ 10 │ │ b ┆ 10 │ │ b ┆ 10 │ │ b ┆ 20 │ └───────┴───────┘ """)  How can I groupby through the fruit key with the constrain of the group having at least 4 values for the sum? So instead of ┌───────┬───────┐ │ fruit ┆ price │ │ --- ┆ --- │ │ str ┆ i64 │ ╞═══════╪═══════╡ │ b ┆ 50 │ │ a ┆ 9 │ └───────┴───────┘  I'd have only fruit b on the output, since it's the only one with at least 4 elements ┌───────┬───────┐ │ fruit ┆ price │ │ --- ┆ --- │ │ str ┆ i64 │ ╞═══════╪═══════╡ │ b ┆ 50 │ └───────┴───────┘ 

Answer: I don't think there's a built-in min_count for this, but you can just filter: ( df.group_by("fruit") .agg(pl.col("price").sum(), pl.len()) .filter(pl.col("len") >= 4) .drop("len") )  shape: (1, 2) ┌───────┬───────┐ │ fruit ┆ price │ │ --- ┆ --- │ │ str ┆ i64 │ ╞═══════╪═══════╡ │ b ┆ 50 │ └───────┴───────┘
Question: Using polars, I am not getting the same output as pandas when calling to_dict. Pandas. df = pd.DataFrame({ 'column_1': [1, 2, 1, 4, 5], 'column_2': ['Alice', 'Bob', 'Alice', 'Tom', 'Tom'], 'column_3': ['Alice1', 'Bob', 'Alice2', 'Tom', 'Tom'] }) df.to_dict(orient='records')  produces [{'column_1': 1, 'column_2': 'Alice', 'column_3': 'Alice1'}, {'column_1': 2, 'column_2': 'Bob', 'column_3': 'Bob'}, {'column_1': 1, 'column_2': 'Alice', 'column_3': 'Alice2'}, {'column_1': 4, 'column_2': 'Tom', 'column_3': 'Tom'}, {'column_1': 5, 'column_2': 'Tom', 'column_3': 'Tom'}]  Polars. df = pl.DataFrame({ 'column_1': [1, 2, 1, 4, 5], 'column_2': ['Alice', 'Bob', 'Alice', 'Tom', 'Tom'], 'column_3': ['Alice1', 'Bob', 'Alice2', 'Tom', 'Tom'] }) df.to_dict(as_series=False)  produces {'column_1': [1, 2, 1, 4, 5], 'column_2': ['Alice', 'Bob', 'Alice', 'Tom', 'Tom'], 'column_3': ['Alice1', 'Bob', 'Alice2', 'Tom', 'Tom']}  Here, the first example is pandas and the output I got when using to_dict with orient='records'. I expected to have the same output in polars. How can I replicate pandas' behaviour in polars?

Answer: To replicate the behaviour of pandas' to_dict with orient="records", you can use pl.DataFrame.to_dicts (notice the extra s). df.to_dicts()  Output. [{'column_1': 1, 'column_2': 'Alice', 'column_3': 'Alice1'}, {'column_1': 2, 'column_2': 'Bob', 'column_3': 'Bob'}, {'column_1': 1, 'column_2': 'Alice', 'column_3': 'Alice2'}, {'column_1': 4, 'column_2': 'Tom', 'column_3': 'Tom'}, {'column_1': 5, 'column_2': 'Tom', 'column_3': 'Tom'}]
Question: I am using  Windows 10 PyCharm 2021.3.3 Professional Edition python 3.11.5 matplotlib 3.8.1  How can I permanently resolve this issue in my development environment? import numpy as np import matplotlib matplotlib.use('Agg') import matplotlib.pyplot as plt # Read data from file, skipping the first row (header) data = np.loadtxt('cm.dat', skiprows=1) # Initialize reference point x0, y0, z0 = data[0] # Compute squared displacement for each time step SD = [(x - x0)**2 + (y - y0)**2 + (z - z0)**2 for x, y, z in data] # Compute the cumulative average of SD to get MSD at each time step MSD = np.cumsum(SD) / np.arange(1, len(SD) + 1) # Generate time steps t = np.arange(1, len(SD) + 1) # Create a log-log plot of MSD versus t plt.figure(figsize=(8, 6)) plt.loglog(t, MSD, marker='o') plt.title('Mean Squared Displacement vs Time') plt.xlabel('Time step') plt.ylabel('MSD') plt.grid(True, which="both", ls="--") plt.show()  C:\Users\pc\AppData\Local\Programs\Python\Python311\python.exe C:/git/RouseModel/tau_plot.py C:\git\RouseModel\tau_plot.py:29: UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown plt.show() Process finished with exit code 0 

Answer: I have the same issue. In my case, I installed the PyQt5==5.15.10. After that, I run my code successfully. pip install PyQt5==5.15.10 or pip install PyQt5 with python==3.11 But from 2024, you guys should install version PyQt6 or the last version with python==3.12 or later.
Question: I'm working with Shapely in Python and trying to subclass the Polygon class. However, I'm encountering an error when trying to add a custom attribute during object creation. Could you please provide guidance on how to subclass the Polygon class in Shapely and add custom attributes without running into this error? This is what I tried so far: from shapely.geometry import Polygon class CustomPolygon(Polygon): def __init__(self, shell=None, holes=None, name=None): super().__init__(shell, holes) self._name = name @property def name(self): return self._name @name.setter def name(self, value): self._name = value polygon1 = CustomPolygon([(0, 0), (0, 1), (1, 1), (1, 0)], name="Polygon1")  And this is the error I get: polygon1 = CustomPolygon([(0, 0), (0, 1), (1, 1), (1, 0)], name="Polygon1") TypeError: __new__() got an unexpected keyword argument 'name' 

Answer: You need to downgrade shapely to version 1.7 unfortunately. They intentionally got rid of any subclassing capability in 1.8 and they explicitly say they aren't going to support it anymore. See the issue here: https://github.com/shapely/shapely/issues/1698
Question: I have a function with a complex return type annotation: from typing import (Union, List) # The -> Union[…] is actually longer than this example: def parse(what: str) -> Union[None, int, bool, float, complex, List[int]]: # do stuff and return the object as needed def parse_from_something(which: SomeType) -> ????: return parse(which.extract_string()) # … class SomeType: def extract_string(self) -> str: # do stuff and return str  How can I type-annotate parse_from_something so that it is annotated to return the same types as parse, without repeating them? The problem I'm solving here is that one function is subject to change, but there's wrappers around it that will always return the identical set of types. I don't want to duplicate code, and because this is a refactoring and after-the-fact type annotation effort, I need to assume I'll remove possible return types from parse in the future, and a static type checker might not realize that parse_from_something can no longer return these.

Answer: What you describe is not possible by design. From PEP 484:  It is recommended but not required that checked functions have annotations for all arguments and the return type. For a checked function, the default annotation for arguments and for the return type is Any. An exception is the first argument of instance and class methods. [...]  So, if you omit the return type, it's Any for external observer. No othe treatment is possible. You have now two options. The first one (preferred, because it gives you type safety and gives your annotations better semantic meaning) is using an alias and just putting it everywhere when you want to say "same type as in X". It obviously scales not only to function return types, but also to class attributes, global constants, function and method arguments, etc. This approach is explained well in @erny answer. Another option is using some trick similar to this answer (but please rename it to snake_case for PEP8 compliance - it's still a function). It introduces a helper decorator to lie a little to your typechecker. When you apply this withParameterAndReturnTypesOf decorator, you basically say: "ignore what happens in function body, treat my func as a twin of the referenced one". It is a beautiful cheat, but this way you don't get the real safety: your implementation may start doing more work and deviate from the contract you signed with the typechecker. E.g. the following will pass: def parse(what: str) -> Union[None, int, bool, float, complex, List[int]]: # do stuff and return the object as needed @dangerous_with_parameters_and_return_type_of(parse) def parse_from_something(which: SomeType): if which is None: return NotImplemented # not covered by the union return parse(which.extract_string())  If you were using the aliased union, you'd get an error from mypy that return type is incompatible.
Question: I have deployed llm model locally which follows openai api schema. As it's endpoint follows openai schema, I don't want to write separate inference client. Is there any way we can utilize existing openai wrapper by langchain to do inference for my localhost model. I checked there is a openai adapter by langchain, but it seems like it require provider, which again I have to write separate client. Overall goal it to not write any redundant code as it's already been maintained by langchain and may change with time. We can modify our api wrt openai and it works out of the box. Your suggestion is appreciated.

Answer: It turns out you can utilize existing ChatOpenAI wrapper from langchain and update openai_api_base with the url where your llm is running which follows openai schema, add any dummy value to openai_api_key can be any random string but is necessary as they have validation for this and finally set model_name to whatever model you've deployed. Rest other params are same as what you'll set in openai, if you want you can set it. from langchain_community.chat_models import ChatOpenAI llm = ChatOpenAI( openai_api_base="http://<host-ip>:<port>/<v1>", openai_api_key="dummy_value", model_name="model_deployed")
Question: I am using seleniumbase with Driver(uc=True), which works well for my specific scraping use case (and appears to be the only driver that consistently remains undetected for me). It is fine for everything that doesn't need specific header settings. For one particular type of scrape I need to set the Request Header (Accept -> application/json). This works fine, and consistently, done manually in Chrome via the Requestly extension, but I cannot work out how to put it in place for seleniumbase undetected Chrome. I tried using execute_cdp_cmd with Network.setExtraHTTPHeaders (with Network.enable first): this ran without error but the request appeared to ignore it. (I was, tbh, unconvinced that the uc=True support was handling this functionality properly, since it doesn't appear to have full Chromium driver capabilities.) Requestly has a selenium Python mechanism, but this has its own driver and I cannot see how it would integrate with seleniumbase undetected Chrome. The built-in seleniumbase wire=True support won't coexist with uc=True, as far as I can see. selenium-requests has an option to piggyback on an existing driver, but this is (to be honest) beyond my embryonic Python skills (though it does feel like this might be the answer if I knew how to put it in place). My scraping requires initial login, so I can't really swap from one driver to another in the course of the scraping session.

Answer: My code fragments from second effective solution derived from now deleted bountied answer (the .v2 was the piece I had not seen previously and which I think is what made it work): ... from seleniumwire import webdriver from selenium.webdriver.chrome.options import Options from seleniumwire.undetected_chromedriver.v2 import Chrome, ChromeOptions ... chrome_options = ChromeOptions() driver = Chrome(seleniumwire_options={'options': chrome_options}) driver.header_overrides = { 'Accept': 'application/json', } ...
Question: I am currently working on a chatbot, and as I am using Windows 11 it does not let me migrate to newer OpenAI library or downgrade it. Could I replace the ChatCompletion function with something else to work on my version? This is the code: import openai openai.api_key = "private" def chat_gpt(prompt): response = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=[{"role": "user", "content": prompt}] ) return response.choices[0].message['content'].strip() if __name__ == "__main__": while True: user_input = input("You: ") if user_input.lower() in ["quit", "exit", "bye"]: break response = chat_gpt(user_input) print("Bot:", response)  And this is the full error:  ... You tried to access openai.ChatCompletion, but this is no longer supported in openai>=1.0.0 - see the README at https://github.com/openai/openai-python for the API. You can run openai migrate to automatically upgrade your codebase to use the 1.0.0 interface. Alternatively, you can pin your installation to the old version, e.g. <pip install openai==0.28> A detailed migration guide is available here: https://github.com/openai/openai-python/discussions/742  I tried both upgrading and downgrading through pip.

Answer: Try updating to the latest and using: from openai import OpenAI client = OpenAI( # defaults to os.environ.get("OPENAI_API_KEY") api_key="private", ) def chat_gpt(prompt): response = client.chat.completions.create( model="gpt-3.5-turbo", messages=[{"role": "user", "content": prompt}] ) return response.choices[0].message.content.strip()  Link EDIT: message.['content'] -> message.content on the return of this function, as a message object is not subscriptable error is thrown while using message.['content']. Also, update link from pointing to the README (subject to change) to migration guide specific to this code.
Question: I use the session to count the number of failed logins. But when I use Wi-Fi internet, because the IP of two devices (mobile and PC) is the same, Django uses the same session. If I have two failed logins on PC and two failed logins on mobile, it will add these two together. But the interesting thing is that when I log into the account from one device, the second device does not log into the account and must be logged into the account separately. That is, it uses two different storage methods to log into the account and count the number of unsuccessful logins. if 'RC' in request.session: VRC = request.session['RC'] else: VRC = 1 VRC += 1 request.session['RC'] = VRC 

Answer: All devices using the same Wi-Fi have the same external IP, what you mentioned as a problem is actually an advantage that prevents malicious attacks on the site.