eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm trying to get my image from S3 bucket and return it. Here's the code: import base64 import boto3 import json import random s3 = boto3.client('s3') def lambda_handler(event, context): number = random.randint(0,1) if number == 1: response = s3.get_object( Bucket='bucket-name', Key='image.png', ) image = response['Body'].read() return { 'headers': { "Content-Type": "image/png" }, 'statusCode': 200, 'body': base64.b64encode(image).decode('utf-8'), 'isBase64Encoded': True } else: return { 'headers': { "Content-type": "text/html" }, 'statusCode': 200, 'body': "<h1>This is text</h1>", }  When I hit my endpoint, an image of a tiny white box is returned. I know image.png exists in my bucket, and when I use the web GUI to open it in my browser, the image is loaded properly. What am I exactly doing wrong here? And in case it matters, here's how I'm uploading the image to S3 (from another Lambda): ... # Prepare image for S3 buffer = io.BytesIO() my_image.save(buffer, 'PNG') buffer.seek(0) # Rewind pointer back to start response = s3.put_object( Bucket=S3_BUCKET_NAME, Key=f'{S3_KEY}{filename}.png', Body=buffer, ContentType='image/png', ) ...  In the above code, my_image is just an image I created using the PIL library. Thanks for any help!

Answer: Here it is how I do this: Your lambda with corrected body: import base64 import boto3 import json import random s3 = boto3.client('s3') def lambda_handler(event, context): response = s3.get_object( Bucket='bucket-name', Key='image.png', ) image = response['Body'].read() return { 'headers': { "Content-Type": "image/png" }, 'statusCode': 200, 'body': base64.b64encode(image), 'isBase64Encoded': True }  API gateway settings  Integration Request
Question: I have a pandas dataframe df = pd.DataFrame([[i+10*j for i in range(6)] for j in range(5)], index=[f"item{i}" for i in range(5)]) df.columns = pd.MultiIndex.from_product((["abc", "xyz"], ["one", "two", "three"]))   abc xyz one two three one two three item0 0 1 2 3 4 5 item1 10 11 12 13 14 15 item2 20 21 22 23 24 25 item3 30 31 32 33 34 35 item4 40 41 42 43 44 45  I cannot figure out how to turn first level of column multiindex into row multiindex to get something like following  one two three item0 abc 0 1 2 xyz 3 4 5 item1 abc 10 11 12 xyz 13 14 15 item2 abc 20 21 22 xyz 23 24 25 item3 abc 30 31 32 xyz 33 34 35 item4 abc 40 41 42 xyz 43 44 45  any hint appreciated. Thank you

Answer: Here you go: df.stack(level=0)
Question: Why am I getting the error? I have added the type properly, right? Invalid index type "str" for "Union[str, Dict[str, str]]"; expected type "Union[int, slice]"  Code from typing import List, Dict, Union d = {"1": 1, "2": 2} listsOfDicts: List[Dict[str, Union[str, Dict[str, str]]]] = [ {"a": "1", "b": {"c": "1"}}, {"a": "2", "b": {"c": "2"}}, ] [d[i["b"]["c"]] for i in listsOfDicts] 

Answer: Mypy expects dictionaries to have the same type. Using Union models a subtype relation, but since Dict type is invariant, the key-value pair must match exactly as defined in the type annotation—which is the type Union[str, Dict[str, str]], so the subtypes in the Union wouldn't get matched (neither str, Dict[str, str] are valid types). To define multiple types for different keys, you should use TypedDict. Usage as seen here: https://mypy.readthedocs.io/en/latest/more_types.html#typeddict. from typing import List, Dict, Union, TypedDict d = {"1": 1, "2": 2} dictType = TypedDict('dictType', {'a': str, 'b': Dict[str, str]}) listsOfDicts: List[dictType] = [ {"a": "1", "b": {"c": "1"}}, {"a": "2", "b": {"c": "2"}}, ] [d[i["b"]["c"]] for i in listsOfDicts]
Question: I'm trying to use regex to match every character between a string and a \r character : text = 'Some text\rText to find !\r other text\r'  I want to match 'Text to find !'. I already tried : re.search(r'Some text\r(.*)\r', text).group(1)  But it gives me : 'Text to find !\r other text' It's surprising because it works perfectly when replacing \r by \n : re.search(r'Some text\n(.*)\n', 'Some text\nText to find !\n other text\n').group(1)  returns Text to find ! Do you know why it behaves differently when we use \r and \n ?

Answer: .* is greedy in nature so it is matching longest match available in: r'Some text\r(.*)\r  Hence giving you: re.findall(r'Some text\r(.*)\r', 'Some text\rText to find !\r other text\r') ['Text to find !\r other text']  However if you change to non-greedy then it gives expected result as in: re.findall(r'Some text\r(.*?)\r', 'Some text\rText to find !\r other text\r') ['Text to find !']   Reason why re.findall(r'Some text\n(.*)\n', 'Some text\nText to find !\n other text\n') gives just ['Text to find !'] is that DOT matches any character except line break and \n is a line break. If you enable DOTALL then again it will match longest match in: >>> re.findall(r'Some text\n([\s\S]*)\n', 'Some text\nText to find !\n other text\n') ['Text to find !\n other text'] >>> re.findall(r'(?s)Some text\n(.*)\n', 'Some text\nText to find !\n other text\n') ['Text to find !\n other text']  Which again changes behavior when you use non-greedy quantifier: re.findall(r'(?s)Some text\n(.*?)\n', 'Some text\nText to find !\n other text\n') ['Text to find !']
Question: Currently the OpenAPI documentation looks like this:  Is it possible to separate it into multiple sections? For example, 2 sections, one being the "books" section that contains the methods from "/api/bookcollection/books/" endpoints and the other containing the endpoints with "/api/bookcollection/authors/". I have consulted the FastApi documentation, but I do not find anything close to the operation I want to do.

Answer: The OpenAPI allows the use of tags to group endpoints. FastAPI also supports this feature. The documentation section can be found here. Example: from fastapi import FastAPI tags_metadata = [ { "name": "users", "description": "Operations with users. The **login** logic is also here.", }, { "name": "items", "description": "Manage items. So _fancy_ they have their own docs.", "externalDocs": { "description": "Items external docs", "url": "https://fastapi.tiangolo.com/", }, }, ] app = FastAPI(openapi_tags=tags_metadata) @app.get("/users/", tags=["users"]) async def get_users(): return [{"name": "Harry"}, {"name": "Ron"}] @app.get("/items/", tags=["items"]) async def get_items(): return [{"name": "wand"}, {"name": "flying broom"}]
Question: I have the following two DataFrames: import pandas as pd df = pd.DataFrame([[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]], index = [0, 0.25, 0.50, 0.75, 1], columns = [0, 0.25, 0.50, 0.75, 1]) df_cross = pd.DataFrame([[0.0, 0.25], [0.0, 0.75], [0.5, 1]], columns = ['indexes_to_keep', 'cols_to_keep'])  df:  0.00 0.25 0.50 0.75 1.00 0.00 0 0 0 0 0 0.25 0 0 0 0 0 0.50 0 0 0 0 0 0.75 0 0 0 0 0 1.00 0 0 0 0 0  df_cross:  indexes_to_keep cols_to_keep 0 0.0 0.25 1 0.0 0.75 2 0.5 1.00  In the df I have my storaged data, and the df_cross contains the indexes and columns that I want to keep the values. The values in df which the index and columns do not match with any row of df_cross I want to replace by a string (for example "NaN"). The expected output is:  0.00 0.25 0.50 0.75 1.00 0.00 NaN 0 NaN 0 NaN 0.25 NaN NaN NaN NaN NaN 0.50 NaN NaN NaN NaN 0 0.75 NaN NaN NaN NaN NaN 1.00 NaN NaN NaN NaN NaN  Thanks in advance.

Answer: Let us try crosstab on df_cross, then use where to mask the values s = pd.crosstab(*df_cross.values.T) df.where(s == 1)    0.00 0.25 0.50 0.75 1.00 0.00 NaN 0.0 NaN 0.0 NaN 0.25 NaN NaN NaN NaN NaN 0.50 NaN NaN NaN NaN 0.0 0.75 NaN NaN NaN NaN NaN 1.00 NaN NaN NaN NaN NaN  PS: pd.crosstab(*df_cross.values.T) is just a syntactical shortcut and is effectively equivalent to using pd.crosstab(df.indexes_to_keep, df.cols_to_keep)
Question: I am interested about training a neural network using JAX. I had a look on tf.data.Dataset, but it provides exclusively tf tensors. I looked for a way to change the dataset into JAX numpy array and I found a lot of implementations that use Dataset.as_numpy_generator() to turn the tf tensors to numpy arrays. However I wonder if it is a good practice, as numpy arrays are stored in CPU memory and it is not what I want for my training (I use the GPU). So the last idea I found is to manually recast the arrays by calling jnp.array but it is not really elegant (I am afraid about the copy in GPU memory). Does anyone have a better idea for that? Quick code to illustrate: import os import jax.numpy as jnp import tensorflow as tf def generator(): for _ in range(2): yield tf.random.uniform((1, )) ds = tf.data.Dataset.from_generator(generator, output_types=tf.float32, output_shapes=tf.TensorShape([1])) ds1 = ds.take(1).as_numpy_iterator() ds2 = ds.skip(1) for i, batch in enumerate(ds1): print(type(batch)) for i, batch in enumerate(ds2): print(type(jnp.array(batch))) # returns: <class 'numpy.ndarray'> # not good <class 'jaxlib.xla_extension.DeviceArray'> # good but not elegant 

Answer: Both tensorflow and JAX have the ability to convert arrays to dlpack tensors without copying memory, so one way you can create a JAX array from a tensorflow array without copying the underlying data buffer is to do it via dlpack: import numpy as np import tensorflow as tf import jax.dlpack tf_arr = tf.random.uniform((10,)) dl_arr = tf.experimental.dlpack.to_dlpack(tf_arr) jax_arr = jax.dlpack.from_dlpack(dl_arr) np.testing.assert_array_equal(tf_arr, jax_arr)  By doing the round-trip to JAX, you can compare unsafe_buffer_pointer() to ensure that the arrays point at the same buffer, rather than copying the buffer along the way: def tf_to_jax(arr): return jax.dlpack.from_dlpack(tf.experimental.dlpack.to_dlpack(arr)) def jax_to_tf(arr): return tf.experimental.dlpack.from_dlpack(jax.dlpack.to_dlpack(arr)) jax_arr = jnp.arange(20.) tf_arr = jax_to_tf(jax_arr) jax_arr2 = tf_to_jax(tf_arr) print(jnp.all(jax_arr == jax_arr2)) # True print(jax_arr.unsafe_buffer_pointer() == jax_arr2.unsafe_buffer_pointer()) # True
Question: I want to write a dataclass definition in Python, but can't refer to that same class inside the declaration. Mainly what I want to achieve is the typing of this nested structure, as illustrated below:  @dataclass class Category: title: str children: [Category] # I can't refer to a "Category" tree = Category(title='title 1', children=[ Category('title 11', children=[]), Category('title 12', children=[]) ]) 

Answer: Option #1 You can wrap class name in a string in order to forward-declare the annotation: from dataclasses import dataclass @dataclass class Category: title: str children: list['Category']  Note: The ability to use list[type] for type hinting in Python was introduced in Python 3.9 as part of PEP 585. For Python versions earlier than 3.9, one can use typing.List instead: from dataclasses import dataclass from typing import List @dataclass class Category: title: str children: List['Category']  Option #2 You can include a __future__ import so that all annotations by default are forward-declared as below. In this case, you can also eliminate the typing import and use new-style annotations in Python 3.7 and above. from __future__ import annotations from dataclasses import dataclass @dataclass class Category: title: str children: list[Category]
Question: I love the pathlib.Path api and use it a lot for quick cli tools. Especially with typer. I have a few tightly related questions:  In UNIX cli commands - is the de facto standard for stdin. Is that the same under Windows? Is there a clean, cross-platform way to have a pathlib.Path object (or actually the {POSIX,Windows}Path it automatically becomes) to represent stdin? And what about stdout? 

Answer: There is no cross-platform pseudo file name for standard input and standard output. In POSIX, it's /dev/stdin for standard input and /dev/stdout for standard output, although many CLI tools would accept - in the command line arguments as a shorthand for standard input or standard output. In Windows, it's CONIN$ for standard input and CONOUT$ for standard output, or simply CON for both standard input and standard output.
Question: I want to include NA values when using groupby() which does not happen by default. I think the option dropna=False make it happen. But when the column is of type Categorical the option has no effect. I assume the best would say there is a well thought design decision behind that. Or maybe it is related to this pandas bug which I do not fully understand? The pandas version I use here is 1.2.5. #!/usr/bin/env python3 import pandas as pd print(pd.__version__) # 1.2.5 # initial data df = pd.DataFrame( { '2019': [1, pd.NA, 0], 'N': [2, 0, 7], } ) print(df) ## groupby()'s working as expected # without NA res = df.groupby('2019').size() print(f'\n{res}') # include NA res = df.groupby('2019', dropna=False).size() print(f'\n{res}') ## now the problems ## convert to Category df['2019'] = df['2019'].astype('category') # PROBLEM: NA is ignored res = df.groupby('2019', dropna=False).size() print(f'\n{res}') # PROBLEM: NA is ignored even observed has no effect res = df.groupby('2019', dropna=False, observed=True).size() print(f'\n{res}')  In the output you see the initial DataFrame first and then two groupby() outputs that behave as expected. But then the last two groupby() outputs ilustrating my problem. 1.2.5 2019 N 0 1 2 1 <NA> 0 2 0 7 2019 0 1 1 1 dtype: int64 2019 0.0 1 1.0 1 NaN 1 dtype: int64 2019 0 1 1 1 dtype: int64 2019 1 1 0 1 dtype: int64 >>> 

Answer: This is a bug. It has been fixed and will be released in pandas 2.0. The simplest workaround is to temporarily undo the categories thing: orig = df['2019'].cat.categories.dtype if np.issubdtype(orig, np.integer) or orig == 'bool': orig = 'Int64' # Allow NA values. res = df.astype({'2019': orig}).groupby('2019', dropna=False, observed=True).size()
Question:  I'm struggling with using python on mac m1, and I found that there's an issue on pipenv for making virtual environment with correct architecture. As you can see on the above picture, when I open the terminal with aram64 architecture and make virtual environment using pipenv, the architecture becomes i386. I'm not sure if this causes a big problem, it blocked me to use some of 3rd party packages such as numpy and pandas, although I failed to reproduce the error. (As I remember, it showed an error message like mach-o: but wrong architecture.) The version of pipenv I'm using is 2021.5.29. > arch arm64 > pipenv --python 3.8 Creating a virtualenv for this project... Pipfile: /Users/seewoolee/development/tmp/Pipfile Using /usr/bin/python3 (3.8.9) to create virtualenv... ⠦ Creating virtual environment...created virtual environment CPython3.8.9.final.0-64 in 388ms creator CPython3macOsFramework(dest=/Users/seewoolee/.local/share/virtualenvs/tmp-miv_sugU, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/seewoolee/Library/Application Support/virtualenv) added seed packages: pip==21.2.4, setuptools==58.1.0, wheel==0.37.0 activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator ✔ Successfully created virtual environment! Virtualenv location: /Users/seewoolee/.local/share/virtualenvs/tmp-miv_sugU Creating a Pipfile for this project... > pipenv shell Launching subshell in virtual environment... . /Users/seewoolee/.local/share/virtualenvs/tmp-miv_sugU/bin/activate > arch i386 

Answer: I had this exact problem, and @Beel's comment gave me the clue that I needed to solve it. In my case, pipenv was referencing a version of python that was built for x86_64. Specifically: $ which pipenv /opt/anaconda3/bin/pipenv $ file /opt/anaconda3/bin/python /opt/anaconda3/bin/python: Mach-O 64-bit executable x86_64  This happened when I installed Anaconda and it put itself at the front of my PATH. After adjusting my path to resolve to the version of Pipenv I had installed, pipenv shell does not switch the arch as a side-effect, and I was able to install the correct versions of numpy, psutil, etc.
Question: I installed python lately on my macos system and when I try to open a python file I see this error popup about Jupyter extension : 

Answer: Just ran into this today (I'm on MacOS for my work computer). In my case, upgrading to the pre-release version of the Jupyter extension (v2022.5.1001281006) solved it right away. If you're not an experienced programmer/software engineer, as is the case for me, I suggest trying to upgrade (or roll back) either VS code or the plugin itself before going with your own dev code for the plugin, as with @CAG2 pt2 answer. (Personally, I had trouble finding a 'product.json' file; perhaps you need an insider's build of VS code for that kind of thing [edit]: https://code.visualstudio.com/api/advanced-topics/using-proposed-api, in which case I think you'd be editing 'package.json').
Question: The classifier script I wrote is working fine and recently added weight balancing to the fitting. Since I added the weight estimate function using 'sklearn' library I get the following error : compute_class_weight() takes 1 positional argument but 3 were given  This error does not make sense per documentation. The script should have three inputs but not sure why it says expecting only one variable. Full error and code information is shown below. Apparently, this is failing only in VS code. I tested in the Jupyter notebook and working fine. So it seems an issue with VS code compiler. Any one notice? ( I am using Python 3.8 with other latest other libraries) from sklearn.utils import compute_class_weight train_classes = train_generator.classes class_weights = compute_class_weight( "balanced", np.unique(train_classes), train_classes ) class_weights = dict(zip(np.unique(train_classes), class_weights)), class_weights  In Jupyter Notebook,  

Answer: After spending a lot of time, this is how I fixed it. I still don't know why but when the code is modified as follows, it works fine. I got the idea after seeing this solution for a similar but slightly different issue. class_weights = compute_class_weight( class_weight = "balanced", classes = np.unique(train_classes), y = train_classes ) class_weights = dict(zip(np.unique(train_classes), class_weights)) class_weights
Question: I installed Jupyter notebook and labs on and EC2 instance and for some reason I get the following error:  ImportError: cannot import name 'filefind' from 'traitlets.utils' (/usr/lib/python3/dist-packages/traitlets/utils/init.py)  Jupyter opens fine in the browser but I can't seem to be able to work in an python notebook.

Answer: I disencourage the solution of op. Downloading and overwriting python libraries is not the way of keeping your system stable and clean! What I found out is that while installing Jupyter notebook it had found four significant errors which resulted from python3 packages that were not installed correctly within that installation itself. ERROR: ipykernel 6.6.0 has requirement traitlets<6.0,>=5.1.0, but you'll have traitlets 4.3.3 which is incompatible. ERROR: jupyterlab-pygments 0.1.2 has requirement pygments<3,>=2.4.1, but you'll have pygments 2.3.1 which is incompatible. ERROR: nbconvert 6.3.0 has requirement pygments>=2.4.1, but you'll have pygments 2.3.1 which is incompatible. ERROR: nbconvert 6.3.0 has requirement traitlets>=5.0, but you'll have traitlets 4.3.3 which is incompatible.  The solution is to just patch the packages to the newest version with: pip3 install traitlets==5.1.1 pip3 install pygments==2.4.1  This applies to all similar cases where outdated packages prevent you from your installation
Question: I want to use mediapipe facedetection module to crop face Images from original images and videos, to build a dataset for emotion recognition. is there a way of getting the bounding boxes from mediapipe faceDetection solution? cap = cv2.VideoCapture(0) with mp_face_detection.FaceDetection( model_selection=0, min_detection_confidence=0.5) as face_detection: while cap.isOpened(): success, image = cap.read() if not success: print("Ignoring empty camera frame.") continue image.flags.writeable = False image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) results = face_detection.process(image) # Draw the face detection annotations on the image. image.flags.writeable = True image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) if results.detections: for detection in results.detections: mp_drawing.draw_detection(image, detection) ## ''' #### here i want to grab the bounding box for the detected faces in order to crop the face image ''' ## cv2.imshow('MediaPipe Face Detection', cv2.flip(image, 1)) if cv2.waitKey(5) & 0xFF == 27: break cap.release()  thank you

Answer: In order to figure out format you can follow two ways: Check protobuf files in medipipe  Check out for what "Detection" is: https://github.com/google/mediapipe/blob/master/mediapipe/framework/formats/detection.proto  We need location_data. It should have format field, which should be BOUNDING_BOX, or RELATIVE_BOUNDING_BOX (but in fact only RELATIVE_BOUNDING_BOX).   Checkout for drawing_utils contents: Just check for draw_detection method. You need line with cv2.rectangle call Here's a snippet  results = face_detection.process(image) # Draw the face detection annotations on the image. image.flags.writeable = True image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR) if results.detections: for detection in results.detections: draw_detection(image, detection) ## ''' #### here i want to grab the bounding box for the detected faces in order to crop the face image ''' ## location_data = detection.location_data if location_data.format == LocationData.RELATIVE_BOUNDING_BOX: bb = location_data.relative_bounding_box bb_box = [ bb.xmin, bb.ymin, bb.width, bb.height, ] print(f"RBBox: {bb_box}")