eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In a multiprocess situation, I want to avoid reading arrays from a zarr group that haven't fully finished writing by the other process yet. This functionality does not seem to come out of the box with zarr. While chunk writing is atomic in zarr, array writing seems not to be (i.e. while you can never have a half-written chunk, you can have a half-written array if said array spans multiple chunks). In my concrete example, one process is writing to the position group. This group contains a 1D array with a chunksize of 100. All goes well if the array I'm writing is smaller than this chunksize. Larger arrays will be written into several chunks, but not all of them are written simultaneously. A parallel process may then try to read the array and find only a first chunk. Zarr then blithely returns an array of 100 elements. Milliseconds later, the 2nd chunk is written, and a subsequent opening of the group now yields 200 elements. I can identify a number of solutions:  A store/group lock which must be acquired before writing or reading the entire array. This works, but makes concurrent writing and reading a lot harder because chunk-level locking is better than group/store-level locking. For simple 1D arrays that are write once/read many, that's enough.  A store/group lock that does not allow reading the entire array while the array is write-locked. I don't know if such read/write locks exist in zarr, or if I should brew my own using the fasteners library. Again for more complex N-D arrays this means loss of performance.  Adjust my write/read code to obtain a lock based on the region to write or read (the lock key could be composed of the indices to write or chunks to write). This would have better performance but it seems absurd that this isn't out-of-the-box supported by zarr.   The zarr docs are a bit too succinct and don't delve very deep into the concept of synchronisation/locking, so maybe I'm just missing something.

Answer: You can't really get around using some form of synchronization. Your best bet is to communicate from the writer process to the reader processes that something is ready to be consumed. A simple multiprocessing.Queue would work if you are forking from the same process and each worker will operate on the array themselves. If multiple workers are coordinating to consume the array in parallel a multiprocessing.Barrier or multiprocessing.Event might work better. There does appear to be some built in synchronization in the zarr.core class, synchronization. Looking at the source code it appears that zarr locks per chunk, so you'll need to do something to coordinate your reads and writes.
Question: I am trying to code an html file that is able to contact with chat gpt. this is the start of the code: <link rel="stylesheet" href="https://pyscript.net/latest/pyscript.css" /> <script defer src="https://pyscript.net/latest/pyscript.js"></script> <py-script> import openai </py-script>  but when I run it it says: ModuleNotFoundError: No module named 'openai'  any idea how I can install it just on the html file? I tried to find out how to use: pip install openai , but I dont know how to run bash commands from the python. I also tried to use <py-env>, but it had no effect on the code whatsoever. + it could be blocked by my proxy, + I'm a beginner to python, so please explain it in simple terms.

Answer: According to the Getting Started documentation, you import external modules using a <py-config> tag: <py-config> packages = ["openai"] </py-config>  However, even with that, this code: <html> <head> <link rel="stylesheet" href="https://pyscript.net/latest/pyscript.css" /> <script defer src="https://pyscript.net/latest/pyscript.js"></script> </head> <body> <py-config> packages = ["openai"] </py-config> <py-script> import openai </py-script> </body> </html>  gives the following error message:  (PY1001): Unable to install package(s) 'openai'. Reason: Can't find a pure Python 3 Wheel for package(s) 'openai'. See: https://pyodide.org/en/stable/usage/faq.html#micropip-can-t-find-a-pure-python-wheel for more information.  At the Pyodide link above is the following:  Why can’t I just use urllib or requests?   We currently can’t use such packages since sockets are not available in Pyodide. See Write http.client in terms of Web APIs for more information.  openai requires requests, so until sockets are implemented in Pyodide (which is used by PyScript), you won't be able to use the openai module, or many others, in PyScript. However, it is possible to make an async HTTP request using PyScript/Pyodide: see this part of the docs.
Question: Is there a way to only show visible values in the legend in plotly express? So for example if this was my original scatter plot:  and I zoom in on a few points like so:  Is there a way to only show the legend values for the points that are currently visible, rather than showing all legend values? Code for reference: fig_2d = px.scatter( proj_2D, x=0, y=1, color=topic_nums.astype(str), labels={'color': 'topic'}, #color_continuous_scale="spectral" ) fig_2d.update_traces(marker=dict(size=5), selector=dict(mode='markers')) fig_2d.update_layout( autosize=False, width=1000, height=800, ) fig_2d.show()  where proj_2D is a 2D matrix

Answer: You can create a JupyterDash app, and use a callback to read in the layout data from zoom events, then update the figure accordingly. In order to reset the default (with all of the points and traces in the legend), you can click Zoom Out tooltip in the figure. import json import numpy as np import pandas as pd import plotly.express as px from jupyter_dash import JupyterDash from dash import dcc, html, Input, Output # Sample Data np.random.seed(42) proj_2D = pd.DataFrame({ 'x': np.random.uniform(low=0.0, high=1.0, size=100), 'y': np.random.uniform(low=0.0, high=1.0, size=100), 'topic': list(range(1,11))*10 }) proj_2D['topic'] = proj_2D['topic'].astype(str) fig_2d = px.scatter( proj_2D, x='x', y='y', color='topic' ) fig_2d.update_traces(marker=dict(size=5), selector=dict(mode='markers')) fig_2d.update_layout( autosize=False, width=1000, height=800, ) # Build App app = JupyterDash(__name__, prevent_initial_callbacks=True) app.layout = html.Div([ html.H1("JupyterDash Scatter Zoom Update"), dcc.Graph(id='scatter-fig', figure=fig_2d), ]) @app.callback( Output('scatter-fig', 'figure'), Input('scatter-fig', 'relayoutData'), ) def display_relayout_data(relayoutData): xaxis_min, xaxis_max = relayoutData["xaxis.range[0]"], relayoutData["xaxis.range[1]"] yaxis_min, yaxis_max = relayoutData["yaxis.range[0]"], relayoutData["yaxis.range[1]"] ## subset data proj_2D_new = proj_2D[ proj_2D['x'].between(xaxis_min, xaxis_max) & proj_2D['y'].between(yaxis_min, yaxis_max) ] fig_2d = px.scatter( proj_2D_new, x='x', y='y', color='topic' ) fig_2d.update_traces(marker=dict(size=5), selector=dict(mode='markers')) fig_2d.update_layout( autosize=False, width=1000, height=800, ) return fig_2d # Run app and display result inline in the notebook app.run_server(mode='inline', debug=True, port=8000)
Question: So, I have an audio file which is very long in duration. I have manual annotations (start and end duration in seconds) of the important parts which I need from the whole audio in a text file. I have converted this text file into a nested list where in each list has [start , end] The whole list looks like [[start1,end1],[start2,end2]......] what I need to do is go through my annotation list shown above, get one timestamp(start and end time sublist) and then crop this part from the whole original audio and then the next timestamp and crop that part out from the whole audio and so on. I understand that I need to make sure the reference for the timings must be in accordance with the first unedited original audio. note that, the timestamps are float values and its quite important to keep them as is. The next step would be to extract audio characteristics such as mfcc from the cropped audio file. fs1, y1 = scipy.io.wavfile.read(file_path) l1 = numpy.array(annotation_list) newWavFileAsList = [] for elem in l1: startRead = elem[0] endRead = elem[1] newWavFileAsList.extend(y1[startRead:endRead]) newWavFile = numpy.array(newWavFileAsList) scipy.io.wavfile.write(sample, fs1, newWavFile)  I have tried it as above, however it shows an error that the indexes startRead and endRead must be integers. I understand referencing y1 using those indexes is completely dumb, but how can I relate the duration which I have in seconds to the indexes of the read audio file? How do you suggest I approach this?

Answer: Try out Pydub! :) from pydub import AudioSegment def trim_audio(intervals, input_file_path, output_file_path): # load the audio file audio = AudioSegment.from_file(input_file_path) # iterate over the list of time intervals for i, (start_time, end_time) in enumerate(intervals): # extract the segment of the audio segment = audio[start_time*1000:end_time*1000] # construct the output file path output_file_path_i = f"{output_file_path}_{i}.wav" # export the segment to a file segment.export(output_file_path_i, format='wav') # test it out print("Trimming audio...") trim_audio([[0, 1], [1, 2]], "test_input.wav", "test_output") print("...done! <3")  This code works for me. Lmk if you encounter any problems. Edit: Just to let you know, I tried this out for floats, and it works just fine. I took a look at it and it seemed like it should behave oddly with floats, but it apparently works just fine. I tried long weird ones like 2.2352344, seems ok. Another edit: I just remembered you might need ffmpeg to be able to use Pydub. To install ffmpeg, go download it, extract it, then add the path of it to your Windows path variable.
Question: Consider the dataframe df_counts, constructed as follows: df2 = pd.DataFrame({ "word" : ["AA", "AC", "AC", "BA", "BB", "BB", "BB"], "letter1": ["A", "A", "A", "B", "B", "B", "B"], "letter2": ["A", "C", "C", "A", "B", "B", "B"] }) df_counts = df2[["word", "letter1", "letter2"]].groupby(["letter1", "letter2"]).count()  Output:  What I would like to do from here, is to order first by letter1 totals, so the rows for letter1 == "B" appear first (there are four words starting with B, vs only three with A), and then ordered within each grouping of letter1 by the values in the word column. So the final output should be:  word letter1 letter2 B B 3 A 1 A C 2 A 1  Is this possible to do?

Answer: When you have a complex sorting order, it's always easy to use numpy.lexsort: # minor sorting order first, major one last # - to inverse the order order = np.lexsort([-df_counts['word'], -df_counts.groupby('letter1')['word'].transform('sum')]) out = df_counts.iloc[order]  The pandas equivalent would be: (df_counts .assign(total=df_counts.groupby('letter1')['word'].transform('sum')) .sort_values(by=['total', 'word'], ascending=False) .drop(columns='total') )  Output:  word letter1 letter2 B B 3 A 1 A C 2 A 1
Question: I am trying to get direct messages from my Mastodon account using Mastodon.py. I am able to get everything on the timeline, but can't seem to return direct messages. Since direct messages are separate from the timeline (on the web interface), I am assuming there is some function other than timeline() I should be using. I looked into the status() function but I would need to already have the ID of the message to use it. I have been searching the API documentation, but there doesn't appear to be any function or method that is obvious to me. My function currently looks like this: def get_direct_messages(mastodon): # Get all status messages from the user's timeline statuses = mastodon.timeline() print(statuses) # Filter the status messages to only include direct messages direct_messages = [status for status in statuses if status["visibility"] == "direct"] print (direct_messages) return direct_messages  This will print out all the messages on the timeline, so I know the connection and everything is valid, but I'm just not doing something right. print(statuses) shows timeline messages only, so I am sure this isn't the right way to access direct messages.

Answer: Instead of mastodon.timeline() you want to call mastodon.conversations()  To get the content of the conversations you can do conversations = mastodon.conversations() for c in conversations: print(c.last_status.content)  https://mastodonpy.readthedocs.io/en/stable/07_timelines.html#mastodon.Mastodon.conversations
Question: I have the following dataset data = {'ID': ['A', 'B', 'C', 'D'], '2012': [0, 1, 1, 1], '2013': [0, 0, 1, 1], '2014': [0, 0, 0, 1], '2015': [0, 0, 1, 1], '2016': [0, 0, 1, 0], '2017': [1, 0, 1,1]} df = pd.DataFrame(data)  For each row I want to generate a new column - Baseline_Year - which assumes the name of the column with all values to the right that are equal to 1. In case there is not column with all the values equal to 1, I would like the Baseline_Year to be equal to missing. See the expected results data = {'ID': ['A', 'B', 'C', 'D', 'E'], '2012': [0, 1, 1, 1, 1], '2013': [0, 0, 1, 1, 1], '2014': [0, 0, 0, 1, 1], '2015': [0, 0, 1, 1, 1], '2016': [0, 0, 1, 0, 1], '2017': [1, 0, 1,1, 1], 'Baseline_Year': [np.nan, np.nan, '2015','2017', '2012'], } df_results = pd.DataFrame(data) df_results 

Answer: I would use a boolean mask and idxmax: # get year columns, identify rightmost 1s m = (df.filter(regex=r'\d+') .loc[:, ::-1] .eq(1).cummin(axis=1) .loc[:, ::-1] ) df['Baseline_Year'] = m.idxmax(axis=1).where(m.any(axis=1))  Output:  ID 2012 2013 2014 2015 2016 2017 Baseline_Year 0 A 0 0 0 0 0 1 2017 1 B 1 0 0 0 0 0 NaN 2 C 1 1 0 1 1 1 2015 3 D 1 1 1 1 0 1 2017  If you want a minimum number of 1s on the right: N = 2 df['Baseline_Year'] = m.idxmax(axis=1).where(m.sum(axis=1).ge(N))  Output:  ID 2012 2013 2014 2015 2016 2017 Baseline_Year 0 A 0 0 0 0 0 1 NaN 1 B 1 0 0 0 0 0 NaN 2 C 1 1 0 1 1 1 2015 3 D 1 1 1 1 0 1 NaN  Intermediate m:  2012 2013 2014 2015 2016 2017 0 False False False False False True 1 False False False False False False 2 False False False True True True 3 False False False False False True
Question: Could someone tell me how I can test an endpoint that uses the new lifespan feature from FastAPI? I am trying to set up tests for my endpoints that use resources from the lifespan function, but the test failed since the dict I set up in the lifespan function is not passed to the TestClient as part of the FastAPI app. My API looks as follows. from fastapi import FastAPI from contextlib import asynccontextmanager ml_model = {} @asynccontextmanager async def lifespan(app: FastAPI): predictor = Predictor(model_version) ml_model["predict"] = predictor.predict_from_features yield # Clean up the ML models and release the resources ml_model.clear() app = FastAPI(lifespan=lifespan) @app.get("/prediction/") async def get_prediction(model_input: str): prediction = ml_model["predict"](model_input) return prediction  And the test code for the /prediction endpoint looks as follows: from fastapi.testclient import TestClient from app.main import app client = TestClient(app) def test_read_prediction(): model_input= "test" response = client.get(f"/prediction/?model_input={model_input}") assert response.status_code == 200  The test failed with an error message saying KeyError: 'predict', which shows that the ml_models dict was not passed with the app object. I also tried using app.state.ml_models = {}, but that didn't work either. I would appreciate any help!

Answer: Use the TestClient as a context manager. This triggers startup/shutdown events as well as lifespans. from fastapi.testclient import TestClient from app.main import app def test_read_prediction(): with TestClient(app) as client: model_input= "test" response = client.get(f"/prediction/?model_input={model_input}") assert response.status_code == 200  Here's some documentation about testing startup/shutdown events. It also applies to lifespans.
Question: I know it isn't a correct thing to do, but I would like to try to install package that requires Python 3.8, but my installed Python is 3.7. Is it possible using pip? Or I must clone the repository and change the setup.py?

Answer: You can use the --ignore-requires-python option. pip install --help   --ignore-requires-python Ignore the Requires-Python information.  You can try it with your package, or also with this minimal setup.py: from setuptools import setup, find_packages setup( name="foobar", version="1.0", packages=find_packages(), python_requires="<3.7" )  With Python 3.7, $ pip install . Processing /home/vvvvv/75726452 ERROR: Package 'foobar' requires a different Python: 3.7.11 not in '<3.7' $ pip install . --ignore-requires-python Processing /home/vvvvv/75726452 Installing collected packages: foobar Running setup.py install for foobar ... done Successfully installed foobar-1.0   However, as Mark Ransom suggested, there probably is a good reason the package you are trying to install does requires a version of Python different from 3.7.
Question: I am new to asyncio and trying to understand basic for loop behavior. The code below executes sequentially, but my naive assumption was that while the sleeps are occurring, other items could be fetched via the for loop and start processing. But that doesn't seem to happen. For example, while the code is "doing something else with 1" it seems like it could fetch the next item from the loop and start working on it while waiting for the sleep to end on item 1. But when I run, it executes sequentially with pauses for the sleeps like a non-async program. What am I missing here? import asyncio class CustomIterator(): def __init__(self): self.counter = 0 def __aiter__(self): return self async def __anext__(self): if self.counter >= 3: raise StopAsyncIteration await asyncio.sleep(1) self.counter += 1 return self.counter async def f(item): print(f"doing something with {item}") await asyncio.sleep(3) async def f2(item): print(f"doing something else with {item}") await asyncio.sleep(2) async def do_async_stuff(): async for item in CustomIterator(): print(f"got {item}") await f(item) await f2(item) if __name__ == '__main__': asyncio.run(do_async_stuff())  Output: got 1 doing something with 1 doing something else with 1 got 2 doing something with 2 doing something else with 2 got 3 doing something with 3 doing something else with 3 

Answer: I think you have a common misunderstanding of how async works. You have written your program to be synchronous. await foo() says to call foo(), and feel free to go do something else while we're waiting for foo to return with its answer. Likewise, getting the next element from your custom iterator says "get the next element of this iterator, but feel free to go do something else while waiting for the result". In both cases, you have nothing else to do, so your code wants. If it is safe for two things in your code to run at once, it is your job to say so, using appropriate primitives.
Question: I am trying to load a large Hugging face model with code like below: model_from_disc = AutoModelForCausalLM.from_pretrained(path_to_model) tokenizer_from_disc = AutoTokenizer.from_pretrained(path_to_model) generator = pipeline("text-generation", model=model_from_disc, tokenizer=tokenizer_from_disc)  The program is quickly crashing after the first line because it is running out of memory. Is there a way to chunk the model as I am loading it, so that the program doesn't crash?  EDIT  See cronoik's answer for accepted solution, but here are the relevant pages on Hugging Face's documentation: Sharded Checkpoints: https://huggingface.co/docs/transformers/big_models#sharded-checkpoints:~:text=in%20the%20future.-,Sharded%20checkpoints,-Since%20version%204.18.0  Large Model Loading: https://huggingface.co/docs/transformers/main_classes/model#:~:text=the%20weights%20instead.-,Large%20model%20loading,-In%20Transformers%204.20.0

Answer: You could try to load it with low_cpu_mem_usage: from transformers import AutoModelForSeq2SeqLM model_from_disc = AutoModelForCausalLM.from_pretrained(path_to_model, low_cpu_mem_usage=True)  Please note that low_cpu_mem_usage requires: Accelerate >= 0.9.0 and PyTorch >= 1.9.0.
Question: I want to Install PySide 2 library But apparently this library is not found. I tried this to install PySide2:  pip3 install PySide2  But after executing this command, I encountered the same problem:  ERROR: Could not find a version that satisfies the requirement PySide2 (from versions: none) ERROR: No matching distribution found for PySide2  I used https://pypi.org/project/PySide2/ to solve my problem but I did not reach any result.

Answer: According to the documentation for versions of python 3.7+ you need to do the following to install the module: pip install pyside6
Question: How can I return all rows where one of the columns passes a given condition, WITHOUT specifying any column? My situation is as follows: import pandas as pd print(pd.__version__) # 1.5.2 x = [ 1, 2, 3 ] y = [ 4, 5, 6 ] z = [ 7, 8, 9 ] df = pd.DataFrame({ "a": x, "b": y, "c": z })  I know you can apply a condition on a given column in order to return all the rows where that column passes the condition. df.loc[df.a > 1] # or df.loc[df["a"] > 1]  Which would output:     a b c     2 5 8   3 6 9     I want to select all rows from the data frame, where any of the columns passes the given condition. df.loc["*" > 1]  Which should output:     a b c     1 4 7   2 5 8   3 6 9     I have tried the following but it sets any non matching values to NaN: df[df > 1]  Output:     a b c     NaN 4 7   2 5 8   3 6 9    

Answer: Use vectorial code with any and boolean indexing: df[df.gt(1).any(axis=1)]
Question: I have tried the obvious way to check my protocol: from typing import Any, Protocol, runtime_checkable @runtime_checkable class SupportsComparison(Protocol): def __eq__(self, other: Any) -> bool: ... issubclass(int, SupportsComparison)  Unfortunately the issubclass() call ends with an exception (Python 3.10.6 in Ubuntu 22.04): $ python3.10 protocol_test.py Traceback (most recent call last): File "protocol_test.py", line 8, in <module> issubclass(object, SupportsComparison) File "/usr/lib/python3.10/abc.py", line 123, in __subclasscheck__ return _abc_subclasscheck(cls, subclass) File "/usr/lib/python3.10/typing.py", line 1570, in _proto_hook raise TypeError("Protocols with non-method members" TypeError: Protocols with non-method members don't support issubclass()  As you can see I added no non-method members to SupportsComparison. Is this a bug in the standard library?

Answer: From the documentation:  A class that overrides __eq__() and does not define __hash__() will have its __hash__() implicitly set to None.  Therefore, in your case, you have the implicit non-method member SupportsComparison.__hash__ = None. You can fix it by declaring __hash__ explicitly: from typing import Any, Protocol, runtime_checkable @runtime_checkable class SupportsComparison(Protocol): def __eq__(self, other: Any) -> bool: ... def __hash__(self) -> int: ... issubclass(int, SupportsComparison)
Question: Apologies, am new to Python so a very basic question. Below is an example line of my method definition. Where would I be using capital Dict/List and lowercase Dict/List? Thanks in advance! Example scenarios below def execute_signals(self, parameter1: dict[list], parameter2: dict) -> list[dict]: def execute_signals(self, parameter1: Dict[List], parameter2: Dict) -> List[Dict]: def execute_signals(self, parameter1: dict[List], parameter2: dict) -> List[dict]: 

Answer: On Python 3.8 and earlier, the name of the collection type is capitalized, and the type is imported from the typing module. Python 3.5 - 3.8 (also works in Python 3.9+): from typing import List, Set, Dict, Tuple x: List[int] = [1] x: Set[int] = {6, 7} x: Dict[str, float] = {"field": 2.0} x: Tuple[int, str, float] = (3, "yes", 7.5) x: Tuple[int, ...] = (1, 2, 3)  Python 3.9+: x: list[int] = [1] x: set[int] = {6, 7} x: dict[str, float] = {"field": 2.0} x: tuple[int, str, float] = (3, "yes", 7.5) x: tuple[int, ...] = (1, 2, 3)  I recommend you read through the mypy's docs about type hints.