eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have multiple JSON files (10 TB ~) on a S3 bucket, and I need to organize these files by a date element present in every json document. What I think that my code needs to do  Read all json files in the s3 bucket. Keep all documents which have the element "creation_date" between 2022-01-01 and 2022-04-01 Save them in another bucket in a parquet format.  I'm not sure that's the right thing to do, considering the size that I'm dealing it. Here's an example of a json document. Each file has multiple of these documents.  { "id": 123456, "creation_date": "2022-01-01T23:35:16", "params": { "doc_info": "AXBD", "return_date": "20/05/2021", "user_name": "XXXXXXXX", "value": "40,00" }, "user_id": "1234567", "type": "TEST" } ]  Here's what I already tried on a DB notebook, but in fact, I can't use the code directly on a notebook. I necessarily need to write a spark code and run on an airflow dag, because I don't have write access on the bucket using directly from the notebook. # Trying to read all the json files df_test = spark.read.json("s3://my-bucket/**/**" + "/*.json") # Filtering all documents that has the creation_date period that I want df_test_filter = df_test.filter(F.col("creation_date").between('2022-01-01','2022-04-01')) # Write parquet on another bucket # In this test, I'm saving on a local bucket that I have write access. df_test_filter.write.mode('overwrite').parquet("s3://my-local-test-bucket/")  That seems to work fine on a single json file that I use to test, but my questions are:  How can I do this without a databricks notebook, and using an airflow dag with pyspark? Thinking in performance issues, there is a better way to do this? 

Answer: Do you want to run the job only once or do you want to do it periodically? One run What you have should work well # Trying to read all the json files sdf = spark.read.json("s3://my-bucket/**/**/*.json")  The only thing I'd add is to partition the output by the date to speed up queries: ( # Filtering all documents that has the creation_date period that I want sdf.filter(F.col("creation_date").between('2022-01-01','2022-04-01')) # Partition by creation date so that's easier to query .partitionBy("creation_date") # Export the data .write.mode('append') .parquet("s3://my-local-test-bucket/") )  Running it periodically Here I wonder what is the file structure is. It is a good idea to have data partitioned by some dates and in this case it looks like you might have the input data partitioned by another date (maybe insert_date?). Assuming that's the case I suggest each day you read that data and then you write it as parquet partitioned by the date you want. This would be done by: # Trying to read all the json files sdf = spark.read.json(f"s3://my-bucket/insert_date={today:%Y-%m-%d}/*/") sdf.partitionBy("creation_date").write.mode('append').parquet("s3://my-local-test-bucket/")  And later on you can simply retrive the data you need with: sdf = ( spark.read.json(f"s3://my-bucket/") .where(F.col("creation_date").between('2022-01-01','2022-04-01')) )
Question: The idea is to get file object from one endpoint and send it to other endpoints to work with it without saving it. Let's have this expample code: import httpx from fastapi import Request, UploadFile, File app = FastAPI() client = httpx.AsyncClient() @app.post("/endpoint/") async def foo(request: Request, file: UploadFile = File(...)) urls = ["/some/other/endpoint", "/another/endpoint/"] for url in urls: response = await client.post(url) # here I need to send the file to the other endpoint return {"bar": "baz"} @app.post("/some/other/endpoint/") async def baz(request: Request, file: UploadFile = File(...)): # and here to use it # Do something with the file object return {"file": file.filename} @app.post("/another/endpoint/") async def baz(request: Request, file: UploadFile = File(...)): # and here to use it too # Do something with the file object return {"file": file.content_type}  As stated here I tried to do something like this: data = {'file': file} response = await client.post(url, data=data)  But it errored with '{"detail":[{"loc":["body","file"],"msg":"Expected UploadFile, received: <class \'str\'>","type":"value_error"}]}'  Example curl request: curl -X 'POST' -F 'file=@somefile' someserver/endpoint/ 

Answer: httpx similar to requests uses files=.... to send files. ie. post(..., files={'file': file.file}, ...)  or with filename post(..., files={'file': (file.filename, file.file)}, ...)   BTW: If you send the same file a few times then you may need to move pointer to the beginning of file after sending file.file.seek(0)  or await file.seek(0)   Full working code from fastapi import FastAPI, Request, UploadFile, File import httpx app = FastAPI() client = httpx.AsyncClient() @app.post("/endpoint/") async def foo(request: Request, file: UploadFile = File(...)): print('/endpoint/') urls = ["/some/other/endpoint/", "/another/endpoint/"] results = [] for url in urls: response = await client.post('http://localhost:8000' + url, files={'file': (file.filename, file.file)}) #file.file.seek(0) # move back at the beginning of file after sending to other URL await file.seek(0) # move back at the beginning of file after sending to other URL results.append(response) results = [item.text for item in results] print('results:', results) return {"bar": "baz"} @app.post("/some/other/endpoint/") async def baz(request: Request, file: UploadFile = File(...)): print('/some/other/endpoint/') print('filename:', file.filename) print('content_type:', file.content_type) # Do something with the file object return {"file": file.filename} @app.post("/another/endpoint/") async def baz(request: Request, file: UploadFile = File(...)): print('/another/endpoint/') print('filename:', file.filename) print('content_type:', file.content_type) # Do something with the file object return {"file": file.content_type}
Question: I want to add some modifications to my force plot (created by shap.plots.force) using Matplotlib, e.g. adding title, using tight layout etc. However, I tried to add title and the title doesn't show up. Any ideas why and how can I add the title using Matplotlib? import numpy as np import shap import matplotlib.pyplot as plt myBaseline=1.5 shap_values_0 = np.array([-1, -4, 3]) test_point_0 = np.array([11, 12, 13]) features_names = ['a1','a2','a3'] shap.plots.force(myBaseline,shap_values_0,test_point_0,features_names,matplotlib = 1) plt.suptitle("This is my title") # It doesn't show up, why? fig = plt.gcf() fig.canvas.draw() plt.close()  

Answer: The last lines in force_plot are: if show: plt.show() else: return plt.gcf()  so, if you set show = False you can get prepared SHAP plot as figure object and customize it to your needs as usual: import shap myBaseline = 1.5 shap_values_0 = np.array([-1, -4, 3]) test_point_0 = np.array([11, 12, 13]) features_names = ["a1", "a2", "a3"] shap.plots.force( myBaseline, shap_values_0, test_point_0, features_names, matplotlib=True, show=False ) plt.title("This is my title", y=1.75) plt.show()
Question: In Python 3.9+ I can write list_of_integers: list[int], but I see senior developers using the older syntax (even in Python 3.9 and 3.10 scripts): from typing import List list_of_integers: List[int]  Is this superior for backwards compatibility and explicitness?

Answer: When the current version of documentation for typing.List says:  Deprecated since version 3.9: builtins.list now supports []. See PEP 585 and Generic Alias Type.  It should be considered best practice unless you have a compelling reason not to use it (like what you said about backward compatibility for older versions of Python).
Question: below is part code: df = None F_DATE = ['202101', '202102', '202103'] for date in F_DATE: if df is None: df = spark.sql("select count(*) as Total_count from test_" + date) else: df2 = spark.sql("select count(*) as Total_count from test_" + date) df = df.union(df2) df.write.csv('/csvs/test.csv')  I tried 'toDF()', 'withColumnRenamed()', and 'selectExpr()', but the column name was not changed. NOTE. Use the table in Hive. ADD I've never used "df.show()" to write code, and I've used "df.show()" to read code. When used "df.show()" in write code, it was confirmed that the column name came out well, and when used "df.show()" in read code, it was confirmed that the column name did not come out properly.

Answer: You can use: df = df.withColumnRenamed('old_name', 'new_name')
Question: I have a sequence continuation/prediction task (input: a sequence of class indices, output: a sequence of class indices) and I use Pytorch. My neural network returns a tensor of shape (batch_size, sequence_length, numb_classes) where the entries are a number proportional to the propability that the class with this index is the next class in the sequence. My targets in the training data are of shape (batch_size, sequence_length) (just the sequences of the real predictions). I want to use the CrossEntropyLoss My question: How do I use the Cross Entropy Loss function? Which input shapes are required? Thank you!

Answer: The documentation page of nn.CrossEntropyLoss clearly states:  Input: shape (C), (N, C) or (N, C, d_1, d_2, ..., d_K) with K >= 1 in the case of K-dimensional loss. Target: If containing class indices, shape (), (N) or (N, d_1, d_2, ..., d_K) with K >= 1 in the case of K-dimensional loss where each value should be between [0, C). If containing class probabilities, the input and each value should be between [0, 1].  Just to be crystal clear, "input" refers to the output prediction of your model while the "target" is the label tensor. In a nutshell, the target must have one less dimension than that of the input. This missing dimension in the target would contain each class logit value. Usually, we say the target is in the dense format, it only contains the class indices corresponding to the true labels. The example you give corresponds to the use case of: #input = (batch_size, sequence_length, numb_classes) #target = (batch_size, sequence_length)  Which is the case of #input = (N, C, d_1) and #target = (N, d_1), i.e;, you need to permute the axes, or tranpose two axes from your input tensor such that it gets a shape of (batch_size, numb_classes, sequence_length) which is (N, C, d_1). You can do so with either torch.Tensor.transpose or torch.Tensor.permute: >>> input.permute(0,2,1)  or >>> input.transpose(1,2)
Question: I am developing web-scraper in playwright and want to upload images to aws-s3 asynchronouslly. but boto3 is not an async function.. how to fix it?  class Boto3: def __init__(self, key, id): self.S3 = boto3.client('s3', aws_access_key_id=aws_key_id, aws_secret_access_key=aws_secret) def upload_stream(self, stream, bucket_name, key): self.S3.put_object(Body=stream, Bucket=bucket_name, Key=key) ... ... class Scraper: def __init__(self, key, id): self.S3 = boto3.client('s3', aws_access_key_id=id, aws_secret_access_key=key) asnyc _save_image(res): buffer = await res.body() # S3.put_object is not an async function. self.S3.put_object( Body=buffer, Bucket=bucket_name, Key=bucket_key, ) async def scrape(): playwright = await async_playwright().start() browser = await playwright.chromium.launch( headless = True, devtools = False ) page = browser.new_page() page.on('response', _save_image) await page.goto('https://www.example.com') scraper = Scraper(key, id) asyncio.run(scraper.scrape())  self.S3.put_object : this func is not an asnyc so want to change it with in async-version. How to fix it? Thanks in advance.

Answer:  How to fix it?  You can't, as boto3 is not async. At best you can try a third party, non-AWS library, such as aioboto3 in place of boto3.
Question: The source code for numpy.interp calls a compiled_interp function which is apparently the interp function imported from numpy.core.multiarray. I went looking for this function but I can not find it inside that file. What am I missing?

Answer: The interp Python function of numpy.core.multiarray is exported in multiarraymodule.c. It is mapped to arr_interp which is a C function defined in compiled_base.c. The heart of the computation can be found here.
Question: Q = [np.array([0, 1]), np.array([1, 2]), np.array([2, 3]), np.array([3, 4])] for q in Q: print(q in Q)  Running the code above, it gives me the result 'True' at the first iteration, while ValueError comes out afterwards.  True   ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()  I have no idea why it starts to go wrong at second iteration. Anybody can help me plz..

Answer: Essentially, you can't use in to test for numpy arrays in a Python list. It will only ever work for the first element, because of an optimisation in the way Python tests for equality. What's happening is that the implementation for list.__contains__ (which in defers to), is using a short-cut to find a match faster, by first checking for identity. Most people using Python know this as the is operator. This is faster than == equality checks because all is has to do is see if the pointers of both objects are the same value, it is worth checking for first. An identity test works the same for any Python object, including numpy arrays. The implementation essentially would look like this if it was written in Python: def __contains__(self, needle): for elem in self: if needle is elem or needle == elem: return True return False  What happens for your list of numpy arrays then is this:  for q in Q, step 1: q = Q[0]  q in Q is then the same as Q.__contains__(Q[0])  Q[0] is self[0] => True!     for q in Q, step 2: q = Q[1]  q in Q is then the same as Q.__contains__(Q[1])  Q[1] is self[0] => False :-( Q[1] == self[0] => array([False, False]), because Numpy arrays use broadcasting to compare each element in both arrays.      The array([False, False]) result is not a boolean, but if wants a boolean result, so it is passed to (C equivalent of) the bool() function. bool(array([False, False])) produces the error you see. Or, done manually: >>> import numpy as np >>> Q = [np.array([0, 1]), np.array([1, 2]), np.array([2, 3]), np.array([3, 4])] >>> Q[0] is Q[0] True >>> Q[1] is Q[0] False >>> Q[1] == Q[0] array([False, False]) >>> bool(Q[1] == Q[0]) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()  You'll have to use any() and numpy.array_equal() to create a version of list.__contains__ that doesn't use (normal) == equality checks: def list_contains_array(lst, arr): return any(np.array_equal(arr, elem) for elem in lst)  and you can then use that to get True for your loop: >>> for q in Q: ... print(list_contains_array(Q, q)) ... True True True True
Question: I have an Enum of days_of_the week in Python: class days_of_the_week(str, Enum): monday = 'monday' tuesday = 'tuesday' wednesday = 'wednesday' thursday = 'thursday' friday = 'friday' saturday = 'saturday' sunday = 'sunday'  I want to access the value using the index. I've tried: days_of_the_week.value[index]  days_of_the_week[index].value  days_of_the_week.values()[index]  and so on... But everything I tried didn't returned me the value of enum (eg. days_of_the_week[1] >>> 'tuesday') Is there a way?

Answer: IIUC, you want to do: from enum import Enum class days_of_the_week(Enum): monday = 0 tuesday = 1 wednesday = 2 thursday = 3 friday = 4 saturday = 5 sunday = 6 >>> days_of_the_week(1).name 'tuesday'
Question: I have an enum : from enum import Enum class MyEnum(Enum): val1 = "val1" val2 = "val2" val3 = "val3"  I would like to validate a pydantic field based on that enum. from pydantic import BaseModel class MyModel(BaseModel): my_enum_field: MyEnum  BUT I would like this validation to also accept string that are composed by the Enum members. So for example : "val1_val2_val3" or "val1_val3" are valid input. I cannot make this field as a string field with a validator since I use a test library (hypothesis and pydantic-factories) that needs this type in order to render one of the values from the enum (for mocking random inputs) So this : from pydantic import BaseModel, validator class MyModel(BaseModel): my_enum_field: str @validator('my_enum_field', pre=True) def validate_my_enum_field(cls, value): split_val = str(value).split('_') if not all(v in MyEnum._value2member_map_ for v in split_val): raise ValueError() return value  Could work, but break my test suites because the field is anymore of enum types. How to keep this field as an Enum type (to make my mock structures still valid) and make pydantic accept composite values in the same time ? So far, I tried to dynamically extend the enum, with no success.

Answer: I looked at this a bit further, and I believe something like this could be helpful. You can create a new class to define the property that is a list of enum values. This class can supply a customized validate method and supply a __modify_schema__ to keep the information present about being a string in the json schema. We can define a base class for generic lists of concatenated enums like this: from typing import Generic, TypeVar, Type from enum import Enum T = TypeVar("T", bound=Enum) class ConcatenatedEnum(Generic[T], list[T]): enum_type: Type[T] @classmethod def __get_validators__(cls): yield cls.validate @classmethod def validate(cls, value: str): return list(map(cls.enum_type, value.split("_"))) @classmethod def __modify_schema__(cls, field_schema: dict): all_values = ', '.join(f"'{ex.value}'" for ex in cls.enum_type) field_schema.update( title=f"Concatenation of {cls.enum_type.__name__} values", description=f"Underscore delimited list of values {all_values}", type="string", ) if "items" in field_schema: del field_schema["items"]  In the __modify_schema__ method I also provide a way to generate a description of which values are valid. To use this in your application: class MyEnum(Enum): val1 = "val1" val2 = "val2" val3 = "val3" class MyEnumList(ConcatenatedEnum[MyEnum]): enum_type = MyEnum class MyModel(BaseModel): my_enum_field: MyEnumList  Examples Models: print(MyModel.parse_obj({"my_enum_field": "val1"})) print(MyModel.parse_obj({"my_enum_field": "val1_val2"}))  my_enum_field=[<MyEnum.val1: 'val1'>] my_enum_field=[<MyEnum.val1: 'val1'>, <MyEnum.val2: 'val2'>]  Example Schema: print(json.dumps(MyModel.schema(), indent=2))  { "title": "MyModel", "type": "object", "properties": { "my_enum_field": { "title": "Concatenation of MyEnum values", "description": "Underscore delimited list of values 'val1', 'val2', 'val3'", "type": "string" } }, "required": [ "my_enum_field" ] }
Question: I have some text in blue #00a2e8, and some text in black on a PNG image (white background). How to remove everything in blue (including text in blue) on an image with Python PIL or OpenCV, with a certain tolerance for the variations of color? Indeed, every pixel of the text is not perfectly of the same color, there are variations, shades of blue. Here is what I was thinking:  convert from RGB to HSV find the Hue h0 for the blue do a Numpy mask for Hue in the interval [h0-10, h0+10] set these pixels to white  Before coding this, is there a more standard way to do this with PIL or OpenCV Python? Example PNG file: foo and bar blocks should be removed 

Answer: Your image has some issues. Firstly, it has a completely superfluous alpha channel which can be ignored. Secondly, the colours around your blues are quite a long way from blue! I used your planned approach and found the removal was pretty poor: #!/usr/bin/env python3 import cv2 import numpy as np # Load image im = cv2.imread('nwP8M.png') # Define lower and upper limits of our blue BlueMin = np.array([90, 200, 200],np.uint8) BlueMax = np.array([100, 255, 255],np.uint8) # Go to HSV colourspace and get mask of blue pixels HSV = cv2.cvtColor(im,cv2.COLOR_BGR2HSV) mask = cv2.inRange(HSV, BlueMin, BlueMax) # Make all pixels in mask white im[mask>0] = [255,255,255] cv2.imwrite('DEBUG-plainMask.png', im)  That gives this:  If you broaden the range, to get the rough edges, you start to affect the green letters, so instead I dilated the mask so that pixels spatially near the blues are made white as well as pixels chromatically near the blues: # Try dilating (enlarging) mask with 3x3 structuring element SE = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (3,3)) mask = cv2.dilate(mask, kernel, iterations=1) # Make all pixels in mask white im[mask>0] = [255,255,255] cv2.imwrite('result.png', im)  That gets you this:  You may wish to diddle with the actual values for your other images, but the principle is the same.
Question: I'm using DRF for a simple API, and I was wondering if there's a way to achieve this behavior:  I've got two models similar to the following:  class Table(models.Model): name = models.CharField(max_length=100) ... class Column(models.Model): original_name = models.CharField(max_length=100) name = models.CharField(max_length=100, blank=True, null=True) ... table = models.ForeignKey(Table, on_delete=models.CASCADE, related_name="columns")   And their serializers as follows:  class ColumnSerializer(serializers.HyperlinkedModelSerializer): table = serializers.HyperlinkedRelatedField( read_only=True, view_name="table-detail" ) class Meta: model = Column fields = ["url", "name", "table"] class TableSerializer(serializers.HyperlinkedModelSerializer): dataset = serializers.HyperlinkedRelatedField( read_only=True, view_name="dataset-detail" ) tags = serializers.SlugRelatedField( many=True, slug_field="name", queryset=Tag.objects.all() ) columns = ColumnSerializer(many=True, read_only=True) class Meta: model = Table fields = [ "url", "name", ... "columns", ]   This returns me an output similar to  { ... "results": [ { "url": "http://0.0.0.0:8001/api/tables/1/", "name": "some-name", "columns": [ { "url": "http://0.0.0.0:8001/api/columns/1/", "name": "id", "table": "http://0.0.0.0:8001/api/tables/1/" }, ... }  which is totally fine. But what I'd really want to do is, if a Column has name=None, it's filtered out from every API ViewSet. I've managed to do it on the ColumnViewSet by doing queryset = queryset.filter(name__isnull=False), but I can't do it for the TableViewSet or others that might show a Column list. I've tried tinkering with the ColumnSerializer, but the best I could get from it was to show nulls on the Column list. I wonder if there's a way of hiding those. EDIT 1: Adding my ViewSets class TableViewSet(viewsets.ModelViewSet): serializer_class = TableSerializer def get_queryset(self): queryset = Table.objects.all().order_by("name") # some query_params filtering return queryset class ColumnViewSet(viewsets.ModelViewSet): serializer_class = ColumnSerializer def get_queryset(self): queryset = Column.objects.all().order_by("id") queryset = queryset.filter(name__isnull=False) # some query_params filtering return queryset 

Answer: You can work with a Prefetch object [Django-doc] to filter the related object collection, so: from django.db.models import Prefetch class TableViewSet(viewsets.ModelViewSet): serializer_class = TableSerializer def get_queryset(self): queryset = Table.objects.prefetch_related( Prefetch('columns', Column.objects.filter(name__isnull=False)) ).order_by('name') # some query_params filtering return queryset
Question: In Haskell there is a simple list function available iterate :: (a -> a) -> a -> [a] iterate f x = x : iterate f (f x)  In python it could be implemented as following: def iterate(f, init): while True: yield init init = f(init)  I was kinda surprised that something basic like this is not part of the functools/itertools modules. Could it be simply costructed in functional style (i.e. without the loop) using the tools provided in these libraries? (Mostly code golf, trying to learn about functional style in Python.)

Answer: You can do it using some of the functions in itertools: from itertools import accumulate, repeat def iterate(func, initial): return accumulate(repeat(None), func=lambda tot, _: func(tot), initial=initial)  Although it's clearly not very clean. Itertools is missing some fundamental functions for constructing streams, like unfoldr. Most of the itertools functions could be defined in terms of unfoldr, as it happens, but functional programming is a little uncomfortable in Python anyways so that might not be much of a benefit.