eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a DataFrame called player: player_df = pd.DataFrame(np.random.rand(10,3), columns=['x','y','more_cols']) _____________________________________________________________________________ player_df: x y more_cols 0 0.352673 0.479360 0.638508 1 0.764669 0.326961 0.778483 2 0.805774 0.911662 0.316030 3 0.114446 0.185147 0.318742 4 0.714803 0.646525 0.084143 5 0.061614 0.837432 0.886669 6 0.179777 0.519559 0.446562 7 0.615326 0.886046 0.581127 8 0.597375 0.196619 0.310331 9 0.670061 0.471363 0.313047  a second DataFrame called checkpoints_df: checkpoints_df = pd.DataFrame(np.random.rand(4,2), columns=['x','y']) checkpoints_df['checkpoint_name'] = ['Alpha', 'Hotel', 'Indigo', 'Papa'] _____________________________________________________________________________ checkpoints_df: x y checkpoint_names 0 0.616945 0.804442 Alpha 1 0.402007 0.556274 Hotel 2 0.478351 0.443920 Indigo 3 0.075494 0.803561 Papa  and a function distance(x1,y1,x2,y2) that calculates the euclidean distance between 2 points, or a list of points and a point (x1 and y1 can be lists). Goal: I want to add a column to 'player_df' for each checkpoint name in 'checkpoints_df['checkpoint_name']', and fill the column with the distance of the player to that checkpoint (this is a mockup problem, the real problem could have a lot of 'checkpoints') So far I've tried many things, but settled on a solution that uses .iterrows(), however this could be slow if there are many checkpoints. This is what I use right now: for _, row in checkpoints_df.iterrows(): player_df[row['name']] = distance(player_df['x'], player_df['y'], row['x'], row['y'])  I've tried using .apply() but couldn't create columns with this. Is there a way to do this without iterating over the second dataframe?

Answer: If your distance function is vectorized, you can do: import numpy as np def distance(x1, y1, x2, y2): return np.sqrt((x2-x1)**2+(y2-y1)**2) tmp = checkpoints_df.set_index('checkpoint_name') for c in checkpoints_df.index: player_df[c] = distance(player_df['x'], player_df['y'], tmp.loc[c, 'x'], tmp.loc[c, 'y'])  Or, fully vectorized: player_df[checkpoints_df['checkpoint_name']] = distance( player_df['x'].to_numpy()[:,None], player_df['y'].to_numpy()[:,None], checkpoints_df['x'].to_numpy()[None], checkpoints_df['y'].to_numpy()[None] )  Output:  x y more_cols Alpha Hotel Indigo Papa 0 0.548814 0.715189 0.602763 0.290325 0.173562 0.538927 0.116871 1 0.544883 0.423655 0.645894 0.448875 0.169807 0.560716 0.204632 2 0.437587 0.891773 0.963663 0.209178 0.323871 0.500542 0.325561 3 0.383442 0.791725 0.528895 0.120166 0.234831 0.404077 0.287810 4 0.568045 0.925597 0.071036 0.339141 0.374280 0.629699 0.311790 5 0.087129 0.020218 0.832620 0.774609 0.660846 0.601313 0.794770 6 0.778157 0.870012 0.978618 0.522455 0.441177 0.800208 0.302696 7 0.799159 0.461479 0.780529 0.619367 0.359296 0.795839 0.243226 8 0.118274 0.639921 0.143353 0.198590 0.345356 0.101950 0.494356 9 0.944669 0.521848 0.414662 0.725433 0.490735 0.930821 0.345899
Question: Got this warning when trying to calculate the intersection between two geometry objects. >>> shapely.intersection(LineString([(0, 0), (1, 1)], LineString([(2.5, 2.5), (3, 3)])) .../lib/python3.9/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection return lib.intersection(a, b, **kwargs) 

Answer: I think the warning occurs when there is no intersection at all between the two geometries, so what I did is to check if there is an intersection first with intersects() and only then calculate the intersection between them. You can then handle the case where no intersection occurs according to your application. def get_iou( polygon1, polygon2): if polygon1.intersects(polygon2): intersect = polygon1.intersection(polygon2).area union = polygon1.union(polygon2).area return intersect/union return 0
Question: I have 4 PyTorch tensors:  data of shape (l, m, n) a of shape (k,) and datatype long b of shape (k,) and datatype long c of shape (k,) and datatype long  I want to slice the tensor data such that it picks the element addressed by a in 0th dimension. In the 1st and 2nd dimensions, I want to pick a patch of values based on the element addressed by b and c. Specifically, I want to pick 9 values - a 3x3 patch around the value addressed by b. Thus my sliced tensor should have a shape (k, 3, 3). MWE: data = torch.arange(200).reshape((2, 10, 10)) a = torch.Tensor([1, 0, 1, 1, 0]).long() b = torch.Tensor([5, 6, 3, 4, 7]).long() c = torch.Tensor([4, 3, 7, 6, 5]).long() data1 = data[a, b-1:b+1, c-1:c+1] # gives error >>> TypeError: only integer tensors of a single element can be converted to an index  Expected output data1[0] = [[143,144,145],[153,154,155],[163,164,165]] data1[1] = [[52,53,54],[62,63,64],[72,73,74]] data1[2] = [[126,127,128],[136,137,138],[146,147,148]] and so on  How can I do this without using for loop? PS:  I've padded data to make sure that the locations addressed by a,b,c are within the limit. I don't need gradients to flow through this operation. So, I can convert these to NumPy and slice if that is faster. But I would prefer a solution in PyTorch. 

Answer: I would first expand the indices and then add shifts to the repeated indices. Note that the shift for the row and column should be reversed. For example, import torch data = torch.arange(200).reshape((2, 10, 10)) a = torch.Tensor([1, 0, 1, 1, 0]).long() b = torch.Tensor([5, 6, 3, 4, 7]).long() c = torch.Tensor([4, 3, 7, 6, 5]).long() index1 = a.repeat_interleave(9) # kernel_size^2 index2 = b.repeat_interleave(9) # kernel_size^2 shift = torch.arange(-1, 2).repeat_interleave(3).repeat(5) # Shape: (kernel_size^2 x 5) -> [-1, -1, -1, 0, 0, 0, 1, 1, 1] shifted_index2 = index2 + shift index3 = c.repeat_interleave(9) shift = torch.arange(-1, 2).repeat(3).repeat(5) # Shape: (kernel_size^2 x 5) -> [-1, 0, 1, -1, 0, 1, -1, 0, 1] shifted_index3 = index3 + shift # Use the indexing arrays to select the patches data1 = data[index1, shifted_index2, shifted_index3].view(5, 3, 3) print(data1[0]) print(data1[1]) print(data1[2])  The output: tensor([[143, 144, 145], [153, 154, 155], [163, 164, 165]]) tensor([[52, 53, 54], [62, 63, 64], [72, 73, 74]]) tensor([[126, 127, 128], [136, 137, 138], [146, 147, 148]])
Question: Consider the following polars dataframes: >>> left = pl.DataFrame(pl.Series('a', [1,5,3,2])) >>> left shape: (4, 1) ┌─────┐ │ a │ │ --- │ │ i64 │ ╞═════╡ │ 1 │ │ 5 │ │ 3 │ │ 2 │ └─────┘ >>> right = pl.DataFrame([pl.Series('a', [0,1,2,3]), pl.Series('b', [4,5,6,7])]) >>> right shape: (4, 2) ┌─────┬─────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═════╪═════╡ │ 0 ┆ 4 │ │ 1 ┆ 5 │ │ 2 ┆ 6 │ │ 3 ┆ 7 │ └─────┴─────┘  I would like to join the two in such a way that the order of the a values from the left dataframe is preserved. A left join seems to do this: >>> left.join(right, on='a', how='left') shape: (4, 2) ┌─────┬──────┐ │ a ┆ b │ │ --- ┆ --- │ │ i64 ┆ i64 │ ╞═════╪══════╡ │ 1 ┆ 5 │ │ 5 ┆ null │ │ 3 ┆ 7 │ │ 2 ┆ 6 │ └─────┴──────┘  My question is: is this behaviour guaranteed? If not, what would be the safe way to do this? I could use with_row_index and then do a final sort, but that seems rather cumbersome. In pandas this can be done concisely with the reindex method.

Answer: Update (November 2024): Polars has decided to no longer guarantee preserving row order in left joins in the future. More background on this decision can be found in this GitHub issue. The bottom line: this guarantee may be expensive. A new maintain_order parameter will be added in the future which allows users to control this behavior. By default, order will not be preserved. In order to future-proof your code now, you can use the workaround mentioned below of adding a row index and sorting on that. Original post: A left join guarantees preserving the order of the left dataframe, at least in the regular engine. For the streaming engine, this might not be guaranteed. If you want to be 'safe', you already have the right workaround in mind to add a row index and sort on that.
Question: I download and clip some youtube videos with pytube but some videos are not downloading and asking for age verification. How can I solve this? Thanks for your advice

Answer: For pytube 15.0.0 I had the AgeRestrictedError in streams contents even using the use_oauth option. I fixed the problem only replacing ANDROID_MUSIC with ANDROID as "client" at line 223 of innertube.py: def __init__(self, client='ANDROID_MUSIC', use_oauth=False, allow_cache=True): def __init__(self, client='ANDROID', use_oauth=False, allow_cache=True):
Question: I've trained a YOLOv5 model and it works well on new images with yolo detect.py I've exported the model to ONNX and now i'm trying to load the ONNX model and do inference on a new image. My code works but I don't get the correct bounding boxes. I need to get the area of the bounding boxes etc. so I can't just use detect.py Also I used Yolo's non_max_suppression to prune the list of bbox but I don't if it's a good solution. What I did yet: # Load image and preprocessing import cv2 import numpy as np img = cv2.imread("image.jpg", cv2.IMREAD_UNCHANGED) resized = cv2.resize(img, (640,640), interpolation = cv2.INTER_AREA).astype(np.float32) resized = resized.transpose((2, 0, 1)) resized = np.expand_dims(resized, axis=0) # Add batch dimension  # run session on ONNX import onnxruntime as ort ort_session = ort.InferenceSession("yolov5.onnx", providers=["CUDAExecutionProvider"]) # compute ONNX Runtime output prediction ort_inputs = {ort_session.get_inputs()[0].name: resized} ort_outs = ort_session.run(None, ort_inputs)  HERE I HAVE TENSOR WITH ALL THE BOUNDING BOXES # Keep only interesting bounding boxes import torch from yolov5.utils.general import non_max_suppression, xyxy2xywh output= torch.from_numpy(np.asarray(ort_outs)) out = non_max_suppression(output, conf_thres=0.2, iou_thres=0.5)[0] # convert xyxy to xywh xyxy = out[:,:4] xywh = xyxy2xywh(xyxy) out[:, :4] = xywh  # Show bbox from PIL import Image, ImageDraw, ImageFont tmp_image = Image.fromarray(img) draw = ImageDraw.Draw(tmp_image) for i,(x,y,w,h,score,class_id) in enumerate(out): real_x = x * w_ratio # resize from model size to image size real_y = y * h_ratio shape = (real_x, real_y, (x + w) * w_ratio, (y + h) * h_ratio) # shape of the bounding box to draw class_id = round(float(class_id)) class_string = list(class_list.keys())[list(class_list.values()).index(class_id)] color = CLASS_RECTANGLE_COLORS[class_string] draw.rectangle(shape, outline=color, width=4) fnt = ImageFont.load_default() draw.multiline_text((real_x + 8, real_y + 8), f"{class_string} {score*100:.2f}%", font=fnt, fill=color) tmp_image.show()  Image with my algorithm vs detect.py : https://i.sstatic.net/ZxNLJ.jpg Can anyone help ?

Answer: I decided to give up and use this code : import cv2 import torch from PIL import Image # Model model = torch.hub.load(path_to_yolo_library, 'custom', path=onnx_path, source='local') img = Image.open(image_path) # PIL image img = img.resize((640,640)) # Inference results = model(img, size=640) # includes NMS # Results results.print() # print results to screen results.show() # display results results.save() # save as results1.jpg, results2.jpg... etc. # Data print('\n', results.xyxy[0]) # print img1 predictions
Question: How can I transform my simple python class like the following into a avro schema? class Testo(SQLModel): name: str mea: int  This is the Testo.schema() output { "title": "Testo", "type": "object", "properties": { "name": { "title": "Name", "type": "string" }, "mea": { "title": "Mea", "type": "integer" } }, "required": [ "name", "mea" ] }  from here I would like to create an Avro record. This can be converted online on konbert.com (select JSON to AVRO Schema) and it results in the Avro schema below. (all valid despite the name field which should be "Testo" instead of "Record".) { "type": "record", "name": "Record", "fields": [ { "name": "title", "type": "string" }, { "name": "type", "type": "string" }, { "name": "properties.name.title", "type": "string" }, { "name": "properties.name.type", "type": "string" }, { "name": "properties.mea.title", "type": "string" }, { "name": "properties.mea.type", "type": "string" }, { "name": "required", "type": { "type": "array", "items": "string" } } ] }  Anyhow, if they can do it, there certainly must be a way to convert it with current python libraries. Which library can do a valid conversion (and also complex python models/classes? If there is an opinion of that this is a wrong approach, that is also welcome - if - pointing out a better way how this translation process can be done.

Answer: It looks like there are a few libraries that aim to provide this kind of functionality:  py-avro-schema has support for generic Python classes dataclasses-avroschema has support for dataclasses, pydantic models, and faust records pydantic-avro requires your Python class to inherit from pydantic.BaseModel
Question: I am making docs for my Python code with mkdocs and mkdocstrings. I would like to have link from the docs to the source code (page and line number on github). Is there a way to automatic add that with function/class syntax (e.g., '::: identifier')? I am looking for something similar to 'SciPy' docs, where they have [source] button (right of function heading). For an example: Poisson distribution

Answer: You can find inspiration in this archived repository: https://github.com/AI2Business/mkdocstrings-sourcelink. As of now (2023/07/06) there's no built-in way to achieve that. VCS support might happen in the future :) (source: am maintainer).
Question: I usually can patch methods of normal objects using pytest monkeypatch. However when I try with a pydantic.BaseModel it fails. from pydantic import BaseModel class Person(BaseModel): name: str age: int def intro(self) -> str: return f"I am {self.name}, {self.age} years old." def test_person_intro(monkeypatch): p = Person(name='Joe', age=20) monkeypatch.setattr(p, 'intro', lambda: 'patched intro') assert p.intro() == 'patched intro'  Raises: monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x10f0d3040> def test_intro(monkeypatch): p = Person(name='Joe', age=20) > monkeypatch.setattr(p, 'intro', lambda: 'patched intro') - - - - - - - - - - - - - - - - - - - - - > ??? E ValueError: "Person" object has no field "intro" pydantic/main.py:422: ValueError 

Answer: You could patch the type: def test_person_intro(monkeypatch): p = Person(name='Joe', age=20) monkeypatch.setattr(Person, 'intro', lambda self: 'patched intro') assert p.intro() == 'patched intro'  Or you could set item in the instance dict: def test_person_intro(monkeypatch): p = Person(name='Joe', age=20) monkeypatch.setitem(p.__dict__, 'intro', lambda: 'patched intro') assert p.intro() == 'patched intro'  The reason an instance setattr isn't working as usual is that pydantic.BaseModel overrides __setattr__ to only allow a restricted set of names. You will see the same error doing person.intro = "other" as well, it is unrelated to pytest's monkeypatch fixture.
Question: I'm able to use the gpt-3.5-turbo-0301 model to access the ChatGPT API, but not any of the gpt-4 models. Here is the code I am using to test this (it excludes my openai API key). The code runs as written, but when I replace "gpt-3.5-turbo-0301" with "gpt-4", "gpt-4-0314", or "gpt-4-32k-0314", it gives me an error openai.error.InvalidRequestError: The model: `gpt-4` does not exist  I have a ChatGPT+ subscription, am using my own API key, and can use gpt-4 successfully via OpenAI's own interface. It's the same error if I use gpt-4-0314 or gpt-4-32k-0314. I've seen a couple articles claiming this or similar code works using 'gpt-4' works as the model specification, and the code I pasted below is from one of them. Is it possible to access the gpt-4 model via Python + API, and if so, how? openai_key = "sk..." openai.api_key = openai_key system_intel = "You are GPT-4, answer my questions as if you were an expert in the field." prompt = "Write a blog on how to use GPT-4 with python in a jupyter notebook" # Function that calls the GPT-4 API def ask_GPT4(system_intel, prompt): result = openai.ChatCompletion.create(model="gpt-3.5-turbo-0301", messages=[{"role": "system", "content": system_intel}, {"role": "user", "content": prompt}]) print(result['choices'][0]['message']['content']) # Call the function above ask_GPT4(system_intel, prompt) 

Answer: Currently the GPT 4 API is restricted, Even to users with a Chat GPT + subscription. You may need to join the Waitlist for the API.
Question: How to play a video with captions in Jupyter notebook? With code snippets from these post, I've tried to play a video inside jupyter notebook:  How can I play a local video in my IPython notebook? how play mp4 video in google colab  from IPython.display import HTML # Show video compressed_path = 'team-rocket.video-compressed.mp4' mp4 = open(compressed_path,'rb').read() data_url = "data:video/mp4;base64," + b64encode(mp4).decode() HTML(""" <video width=400 controls> <source src="%s" type="video/mp4"> </video> """ % data_url)  [out]:  When I tried to add a .vtt file as caption, the option appears from IPython.display import HTML # Show video compressed_path = 'team-rocket.video-compressed.mp4' mp4 = open(compressed_path,'rb').read() data_url = "data:video/mp4;base64," + b64encode(mp4).decode() HTML(""" <video width=400 controls> <source src="%s" type="video/mp4"> <track src="team-rocket.vtt" label="English" kind="captions" srclang="en" default > </video> """ % data_url)  [out]:  But the subtitles/captions are not present when playing the video. How to play a video with captions in Jupyter notebook? The files used in the examples above are on:  team-rocket.vtt team-rocket.video-compressed.mp4 file can be found on https://colab.research.google.com/drive/1IHgo9HWRc8tGqjmwWGunzxXDVhPaGay_?usp=sharing 

Answer: Try this: from IPython.display import HTML from base64 import b64encode video_path = 'team-rocket.video-compressed.mp4' captions_path = 'team-rocket.vtt' with open(video_path, 'rb') as f: video_data = f.read() video_base64 = b64encode(video_data).decode() with open(captions_path, 'r') as f: captions_data = f.read() captions_base64 = b64encode(captions_data.encode('utf-8')).decode() video_html = f""" <video width="640" height="360" controls> <source src="data:video/mp4;base64,{video_base64}" type="video/mp4"> <track src="data:text/vtt;base64,{captions_base64}" kind="captions" srclang="en" label="English" default> </video> """ HTML(video_html)  For some reason, explicitly streaming in the video + captions/subtitle while specifying the encoding would bypass the security issues that @igrinis' answer pointed out.  "data:video/mp4;base64,{video_base64}" "data:text/vtt;base64,{captions_base64}"
Question: Consider the following example zz = pl.DataFrame({'group' : ['a','a','a','a','b','b','b'], 'col' : [1,2,3,4,1,3,2]}) zz Out[16]: shape: (7, 2) ┌───────┬─────┐ │ group ┆ col │ │ --- ┆ --- │ │ str ┆ i64 │ ╞═══════╪═════╡ │ a ┆ 1 │ │ a ┆ 2 │ │ a ┆ 3 │ │ a ┆ 4 │ │ b ┆ 1 │ │ b ┆ 3 │ │ b ┆ 2 │ └───────┴─────┘  I am trying to create a binned variable by group, essentially replicating a pandas qcut by group. This is easy in Pandas, as shown here: xx = pl.DataFrame({'group' : ['a','a','a','a','b','b','b'], 'col' : [1,2,3,4,1,3,2]}).to_pandas() xx.groupby('group').col.transform(lambda x: pd.qcut(x, q = 2, labels = False)) Out[18]: 0 0 1 0 2 1 3 1 4 0 5 1 6 0 Name: col, dtype: int64  But how to do this in Polars? Thanks!

Answer: Update: Series.qcut was added in polars version 0.16.15 As it's not available on expressions as of yet, you could .partition_by pl.concat( frame.get_column("col") .qcut([.5], maintain_order=True) .select(pl.col("category").to_physical()) for frame in df.partition_by("group") )  shape: (7, 1) ┌──────────┐ │ category │ │ --- │ │ u32 │ ╞══════════╡ │ 0 │ │ 0 │ │ 1 │ │ 1 │ │ 0 │ │ 1 │ │ 0 │ └──────────┘  Or .apply in a groupby context: df.with_columns( pl.col("col") .apply( lambda x: x.qcut([.5], maintain_order=True)["category"].to_physical()) .over("group") )  shape: (7, 2) ┌───────┬─────┐ │ group ┆ col │ │ --- ┆ --- │ │ str ┆ u32 │ ╞═══════╪═════╡ │ a ┆ 0 │ │ a ┆ 0 │ │ a ┆ 1 │ │ a ┆ 1 │ │ b ┆ 0 │ │ b ┆ 1 │ │ b ┆ 0 │ └───────┴─────┘
Question: I have trained a YOLOv8 object detection model using a custom dataset, and I want to convert it to a Core ML model so that I can use it on iOS. After exporting the model, I have a converted model to core ml, but I need the coordinates or boxes of the detected objects as output in order to draw rectangular boxes around the detected objects. As a beginner in this area, I am unsure how to achieve this. Can anyone help me with this problem? Training model: !yolo task=detect mode=train model=yolov8s.pt data= data.yaml epochs=25 imgsz=640 plots=True  Validation: !yolo task=detect mode=val model=runs/detect/train/weights/best.pt data=data.yaml  Export this model to coreML: !yolo mode=export model=runs/detect/train/weights/best.pt format=coreml  How can I get the co ordinate output?

Answer: To get the cordinates as output use nms=True from ultralytics import YOLO model=YOLO('best.pt') model.export(format='coreml',nms=True)  or yolo export model=path/to/best.pt format=onnx nms=True  This will give a option to preview your model in Xcode , and the output will return coordinates
Question: I'm trying to use OpenAI's open source Whisper library to transcribe audio files. Here is my script's source code: import whisper model = whisper.load_model("large-v2") # load the entire audio file audio = whisper.load_audio("/content/file.mp3") #When i write that code snippet here ==> audio = whisper.pad_or_trim(audio) the first 30 secs are converted and without any problem they are converted. # make log-Mel spectrogram and move to the same device as the model mel = whisper.log_mel_spectrogram(audio).to(model.device) # detect the spoken language _, probs = model.detect_language(mel) print(f"Detected language: {max(probs, key=probs.get)}") # decode the audio options = whisper.DecodingOptions(fp16=False) result = whisper.decode(model, mel, options) # print the recognized text if available try: if hasattr(result, "text"): print(result.text) except Exception as e: print(f"Error while printing transcription: {e}") # write the recognized text to a file try: with open("output_of_file.txt", "w") as f: f.write(result.text) print("Transcription saved to file.") except Exception as e: print(f"Error while saving transcription: {e}")  In here: # load the entire audio file audio = whisper.load_audio("/content/file.mp3")  when I write below: " audio = whisper.pad_or_trim(audio) ", the first 30 secs of the sound file is transcribed without any problem and language detection works as well, but when I delete it and want the whole file to be transcribed, I get the following error:  AssertionError: incorrect audio shape  What should I do? Should I change the structure of the sound file? If yes, which library should I use and what type of script should I write?

Answer: I had the same problem and after some digging I found that whisper.decode is meant to extract metadata about the input, such as the language, and hence the limit to 30 seconds. (see source code for decode function here) In order to transcribe (even audio longer than 30 seconds) you can use whisper.transcribe as shown in the following snippet import whisper model = whisper.load_model("large-v2") # load the entire audio file audio = whisper.load_audio("/content/file.mp3") options = { "language": "en", # input language, if omitted is auto detected "task": "translate" # or "transcribe" if you just want transcription } result = whisper.transcribe(model, audio, **options) print(result["text"])  You can find some documentation of the transcribe method in the source code along with some documentation about the DecodingOptions structure