eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm working with the following snippet: from cape_privacy.pandas.transformations import Tokenizer max_token_len = 5 @pandas_udf("string") def Tokenize(column: pd.Series)-> pd.Series: tokenizer = Tokenizer(max_token_len) return tokenizer(column) spark_df = spark_df.withColumn("name", Tokenize("name"))  Since Pandas UDF only uses Pandas series I'm unable to pass the max_token_len argument in the function call Tokenize("name"). Therefore I have to define the max_token_len argument outside the scope of the function. The workarounds provided in this question weren't really helpful. Are there any other possible workarounds or alternatives to this issue? Please Advise

Answer: After trying a myriad of approaches, I found an effortless solution as illustrated below: I created a wrapper function (Tokenize_wrapper) to wrap the Pandas UDF (Tokenize_udf) with the wrapper function returning the Pandas UDF's function call. def Tokenize_wrapper(column, max_token_len=10): @pandas_udf("string") def Tokenize_udf(column: pd.Series) -> pd.Series: tokenizer = Tokenizer(max_token_len) return tokenizer(column) return Tokenize_udf(column) df = df.withColumn("Name", Tokenize_wrapper("Name", max_token_len=5))  Using partial functions (@Vaebhav's answer) did actually make this issue's implementation difficult.
Question: I'm having a difficult time understanding what's happening in these two examples I found of the Kadane Algorithm. I'm new to Python and I'm hoping understanding this complex algo will help me see/read programs better. Why would one example be better than the other, is it just List vs Range? Is there something else that makes one of the examples more efficient? Also, some questions about what's happening in the calculations. (questions inside the examples) I've used PythonTutor to help me get a visual on what exactly is happening step by step. Example 1: In PythonTuter, when you select next step in the screen shot provided, The value of so_far turns to 1. How is this? Giving the sum, I've thought its adding -2 + 1 which is -1, so when so_far turns to 1, how is this?  def max_sub(nums): max_sum = 0 so_far = nums[0] for x in nums[1:]: so_far = max(x, x + so_far) max_sum = max(so_far, max_sum) return max_sum nums = [-2, 1, -3, 4, -1, 2, 1, -5, 4] max_sub(nums) 6   Example 2: Similar question for this one, when I select NEXT step, the max_sum turns from -2 to 4... but how so if it's adding the element in the 2 (which is 4). To me, that would be -2 + 4 = 2 ? def maxSubArraySum(a,size): max_so_far =a[0] curr_max = a[0] for i in range(1,size): curr_max = max(a[i], curr_max + a[i]) max_so_far = max(max_so_far,curr_max) return max_so_far a = [-2, -3, 4, -1, -2, 1, 5, -3] print("Maximum contiguous sum is" , maxSubArraySum(a,len(a))) Maximum contiguous sum is 7   So, this would be a 2 part question than: [1]Based on understandings, why would one be more pythonic and more efficient than the other? [2]How can I better understand the calculations happening in the examples? 

Answer: Simply watch each step and you could figure out this problem: [Notes] this program seems to work based on the assumption of mixed integer numbers? only positive and negatives. # starting so_far = -2 # init. to nums[0] max_sum = 0 # in the for-loop: x = 1 # starting with nums[1:] so_far = max(1, -1) -> 1 (x is 1, -2 + 1) max_sum = max(0, 1) -> 1 ..... continue .... each step is to find the max accumulated numbers sum, as it's evident in the max( ) statement. *There is no `sum` involved, except it tried to determine the current x is good (not negative) then so add it to the so_far.
Question: class mail(BaseModel): mailid: int email: str class User(BaseModel): id: int name: str mails: List[mail] data1 = { 'id': 123, 'name': 'Jane Doe', 'mails':[ {'mailid':1,'email':'aeajhs@gmail.com'}, {'mailid':2,'email':'aeajhsds@gmail.com'} ] } userobj = User(**data1) # Accepted data2 = { 'id': 123, 'name': 'Jane Doe', 'mails':[ {'mailid':1,'email':'aeajhs@gmail.com'}, {'email':'aeajhsds@gmail.com'} ] } userobj = User(**data2) # Discarded or not accepted  I want to check the keys in the dictionary that we passing to pydantic model so If the key is not present in the given dictionary I want to discard that data. For example in data2 in mails {'email':'aeajhsds@gmail.com'} data2 must be discarded

Answer: You may use pydantic.validator as @juanpa-arrivillaga said. There are few little tricks:  Optional it may be empty when the end of your validation. pre=True whether or not this validator should be called before the standard validators (else after)  from pydantic import BaseModel, validator from typing import List, Optional class Mail(BaseModel): mailid: int email: str class User(BaseModel): id: int name: str mails: Optional[List[Mail]] @validator('mails', pre=True) def mail_check(cls, v): mail_att = [i for i in Mail.__fields__.keys()] mail_att_count = 0 for i, x in enumerate(v): for k in dict(x).keys(): if k in mail_att: mail_att_count += 1 if mail_att_count != len(mail_att): v.pop(i) mail_att_count = 0 return v  data = { 'id': 123, 'name': 'Jane Doe', 'mails':[ {'mailid':1,'email':'aaa@gmail.com'}, {'mailid':2,'email':'bbb@gmail.com'}, {'email':'ccc@gmail.com'} ] } x = User(**data) # Discarded or not accepted print(x.id) print(x.name) print(x.mails) # Output # >>123 # >>Jane Doe # >>[Mail(mailid=1, email='aaa@gmail.com'), Mail(mailid=2, email='bbb@gmail.com')]
Question: I cant activate virtual env in vs code. I tried same code in the cmd console is work but not in the vs code terminal. "D:\python\djangoapp\djangovenv\Scripts\activate.bat" I write this code. I am using windows 10 pro

Answer: yeah Its beacuse of terminal vs code was using powershell ı changed with cmd
Question: I needed to remove the characters in string1 which are present in string2. Here string1 and string2 have only the lower case characters a-z with given condition that the length of string1 will be greater every time. I was using the in operator: def removeChars (string1, string2): for char in string2: if char in string1: string1 = string1.replace(char, '') return string1  But I read one answer on Stack Overflow which says:  For container types such as list, tuple, set, frozenset, dict, or collections.deque, the expression x in y is equivalent to any(x is e or x == e for e in y).  Which means the in operator is using a for loop behind the scenes. So my question is, in the for loop in my code, should I consider that nested for loops are being used, as the in operator is using a for loop in the background? If yes, what would be the time complexity of this program?

Answer: in does not necessarily use loops behind the scenes. For example: r = range(100000000000) print(333 in r) # prints True immediately without looping  If you were to loop r it will take quite a long time, so clearly that doesn't happen. in basically calls (behind the scenes) the object's __contains__ method. For some iterators it will in fact "loop" through everything but that's not always the case. This example is basically the same as calling: r.__contains__(333)  As pointed out in comments - the str objects specifically have a smarter algorithm than just plain loops, as you can see here Also see example answer here And see the documentation here Because the real world scenario would probably mean that string1 can be arbitrarily long, but the characters to be removed would be a finite and small set, it will probably be much more efficient to just add up all the characters that aren't in string2. Something like this: def removeChars (string1, string2): result = '' for char in string1: if char not in string2: result += char return result  This will involve looping over string1 just once, but multiple checks against string2 using in. This can be further simplified (to avoid += loop over the result): def removeChars (string1, string2): return ''.join(char for char in string1 if char not in string2)
Question: I want to store metadata for my ML models in pydantic. Is there a proper way to access a fields type? I know you can do BaseModel.__fields__['my_field'].type_ but I assume there's a better way. I want to make it so that if a BaseModel fails to instantiate it is very clear what data is required to create this missing fields and which methods to use. Something like this : from pydantic import BaseModel import pandas as pd # basic model class Metadata(BaseModel): peaks_per_day: float class PeaksPerDayType(float): data_required = pd.Timedelta("180D") data_type = "foo" @classmethod def determine(cls, data): return cls(data) # use our custom float class Metadata(BaseModel): peaks_per_day: PeaksPerDayType def get_data(data_type, required_data): # get enough of the appropriate data type return [1] # Initial data we have metadata_json = {} try: metadata = Metadata(**metadata_json) # peaks per day is missing except Exception as e: error_msg = e missing_fields = error_msg.errors() missing_fields = [missing_field['loc'][0] for missing_field in missing_fields] # For each missing field use its type hint to find what data is required to # determine it and access the method to determine the value new_data = {} for missing_field in missing_fields: req_data = Metadata[missing_field].data_required data_type = Metadata[missing_field].data_type data = get_data(data_type=data_type, required_data=req_data) new_data[missing_field] = Metadata[missing_field].determine(data) metadata = Metadata(**metadata_json, **new_data) 

Answer: In the case you dont need to handle nested classes, this should work from pydantic import BaseModel, ValidationError import typing class PeaksPerDayType(float): data_required = 123.22 data_type = "foo" @classmethod def determine(cls, data): return cls(data) # use our custom float class Metadata(BaseModel): peaks_per_day: PeaksPerDayType def get_data(data_type, required_data): # get enough of the appropriate data type return required_data metadata_json = {} try: Metadata(**metadata_json) except ValidationError as e: field_to_type = typing.get_type_hints(Metadata) missing_fields = [] for error in e.errors(): if error['type']=='value_error.missing': missing_fields.append(error['loc'][0]) else: raise new_data = {} for field in missing_fields: type_ = field_to_type[field] new_data[field] = get_data(type_.data_type, type_.data_required) print(Metadata(**metadata_json, **new_data)) peaks_per_day=123.22  Im not really sure whats the point of data_type or get_data, but I assume its some internal logic that you want to add
Question: Processes are mutating things they should not be able to mutate. A Workerhas a single state variable (an mp.Value). This value is set to -1, and it (the Worker) changes it to 1 in a loop. However, it seems to be possible to reset that value back to -1 by spawning a second Worker, even though this shares nothing with the original pair. This seems like it should be impossible. Behavior: When the second Worker spins up, the state of the first worker (self.state.value ) gets reset to -1. This gets caught, and we print out that an error was discovered. Code: import multiprocessing as mp import time class Worker: def __init__(self, tag, service_state) -> None: self.tag = tag self.local_state = int(service_state.value) self.state = service_state self.run_work_loop() def run_work_loop(self) -> None: print(f"[{self.tag}] Running... {self.state.value} {self.local_state}") while True: if self.state.value != self.local_state: print(f"[{self.tag}] Illegal change. Shared state: {self.state.value} Local State: {self.local_state}") break elif self.state.value == -1: self.state.value = self.local_state = 1 print(f"[{self.tag}] Set Shared State: {self.state.value} Local State: {self.local_state}.") if __name__ == "__main__": mp.Process(target=Worker, args=("A", mp.Value('i', -1))).start() time.sleep(.03) mp.Process(target=Worker, args=("B", mp.Value('i', -1))).start()  Output: [A] Running... -1 -1 [A] Set Shared State: 1 Local State: 1. [A] Illegal change. Shared state: -1 Local State: 1 [B] Running... -1 -1 [B] Set Shared State: 1 Local State: 1. 

Answer: The issue is that you are creating Value instances that immediately go out of scope in the parent process, which makes them get garbage collected. Because of the way Python allocates memory for multiprocessing.Value objects, the second Value ends up using the exact same shared memory location as the first Value, which means the second ends up stomping on the first. You can do some experiments to see this in action. For example, this does not print the warning: if __name__ == "__main__": mp.Process(target=Worker, args=("A", mp.Value('i', -1))).start() time.sleep(.03) a = mp.Value('i', 1) mp.Process(target=Worker, args=("B", mp.Value('i', -1))).start()  The Value we assign to a is initialized to 1, which overwrites the anonymous Value we passed to process "A". Because we overwrite it with 1, no illegal state message is printed. if we instead initialize it to any other value, you will see the warning again. This prints an illegal state message about -2, , for example: if __name__ == "__main__": mp.Process(target=Worker, args=("A", mp.Value('i', -1))).start() time.sleep(.03) a = mp.Value('i', -2) mp.Process(target=Worker, args=("B", mp.Value('i', -1))).start()  Your code should really save the Value instances you create as local variables in your parent process, both to avoid this issue, and because it's pointless to create shared values that you don't actually share. Like this: if __name__ == "__main__": a = mp.Value('i', -1) mp.Process(target=Worker, args=("A", a)).start() time.sleep(.03) b = mp.Value('i', -1) mp.Process(target=Worker, args=("B", b)).start()
Question: I am writing a pandas dataframe to google sheets using gspread: from gspread_formatting import * import gspread from df2gspread import df2gspread as d2g import pandas as pd d2g.upload(data, sheet.id, 'test_name', clean=True, credentials=creds, col_names=True, row_names=False)  While the pandas dataframe is rounded to 2 decimal points the value in google sheets sometimes have more than that. df a b 100.56 600.79  Result in google sheets: aa b 100.5616 600.79  And I can't find any information how I can round a value using python gspread API.

Answer: If you get the worksheet element you can use format to achieve what you want. sh = gc.open("sheet_name") worksheet = sh.get_worksheet(0) # your sheet number worksheet.format('A', {'numberFormat': {'type' : 'NUMBER', 'pattern': '0.0#'}})
Question: I'm trying to set up a FastAPI server that can receive a single file upload from the command line using curl. I'm following the FastAPI Tutorial here: https://fastapi.tiangolo.com/tutorial/request-files/?h=upload+file from typing import List from fastapi import FastAPI, File, UploadFile from fastapi.responses import HTMLResponse app = FastAPI() @app.post("/file/") async def create_file(file: bytes = File(...)): return {"file_size": len(file)} @app.post("/uploadfile/") async def create_upload_file(file: UploadFile = File(...)): return {"filename": file.filename} @app.post("/files/") async def create_files(files: List[bytes] = File(...)): return {"file_sizes": [len(file) for file in files]} @app.post("/uploadfiles/") async def create_upload_files(files: List[UploadFile] = File(...)): return {"filenames": [file.filename for file in files]}  Running this code and then opening "http://127.0.0.1:5094" in a browser gives me a upload form with four ways of selecting files and uploading I followed this tutorial: https://medium.com/@petehouston/upload-files-with-curl-93064dcccc76 I tried uploading a file "1.json" in the current directory like this curl -F "file=@1.json" http://127.0.0.1:5094/uploadfiles  on the server side I get this result INFO: 127.0.0.1:58772 - "POST /uploadfiles HTTP/1.1" 307 Temporary Redirect  I do not understand why a redirect happens. I need help on how to either guess the correct curl syntax or fix this on the FastAPI side.

Answer: The solution was to tell curl to follow a redirect. curl -L -F "file=@1.json" http://127.0.0.1:5094/uploadfile  which then uploads the file.
Question: According to namecheap api docs, a request should have this structure: response_request = f'https://api.namecheap.com/xml.response?ApiUser={ApiUser}&ApiKey={ApiKey}&UserName={ApiUser}&Command=namecheap.domains.check&ClientIp={ClientIp}&DomainList={DomainList}'  But I keep receiving Error Number="1011150" Invalid request IP when I input ClientIp as my IP address (I use a shared IP).

Answer: The ClientIp is:  The public IP address of the system making the request. Google search for "What is my IP" for several services that will provide your public IP address. The same public IP address must be whitelisted. This link provides details on Whitelisting IP.
Question: I've looked all over for some supported library that does this but can't find anything. I just want to run my lambda as a local api (ie localhost:80000/api/get/1) so I can run both my frontend and backend all on my machine for rapid development. I've hacked together a fastapi "gateway" that I run locally and use that to call the lambda_entry locally, the only problem is it's quite slow, no doubt the spinning up of an environment for each request is taxing on performance. I feel like this is something people would use a lot, am I on the right track?

Answer: You can use AWS SAM for this.  Local Testing and Debugging Use SAM CLI to step-through and debug your code. It provides a Lambda-like execution environment locally and helps you catch issues upfront.  You might need to install Docker first as it would be the execution environment used to run the APIs. Setup the sam project first. $ python3 -m pip install aws-sam-cli $ sam init # Just choose the first options e.g. <1 - AWS Quick Start Templates>  Now, you would have a file structure like this: $ tree . └── sam-app ├── events │ └── event.json ├── hello_world │ ├── app.py │ ├── __init__.py │ └── requirements.txt ├── __init__.py ├── README.md ├── template.yaml └── tests ├── __init__.py ├── integration │ ├── __init__.py │ └── test_api_gateway.py ├── requirements.txt └── unit ├── __init__.py └── test_handler.py 6 directories, 13 files  2 files that you would be interested at are: sam-app/hello_world/app.py  This is where you would put the codes of your Lambda function.  import json def lambda_handler(event, context): """Sample pure Lambda function""" return { "statusCode": 200, "body": json.dumps({ "message": "hello world", }), }  sam-app/template.yaml  This is where you would configure the API e.g. HTTP Method, URL, the location of the code to run, etc.  ... HelloWorldFunction: Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction Properties: CodeUri: hello_world/ Handler: app.lambda_handler Runtime: python3.9 Events: HelloWorld: Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api Properties: Path: /hello Method: get ...  Build the app and then you can run your local API already $ cd sam-app/ $ sam build $ sam local start-api  Access your local API  Note that running this the first time might take some time as this is when the environment for the API is setup e.g. creating the image and running the container. Any subsequent calls should be fast already.  $ curl http://127.0.0.1:3000/hello {"message": "hello world"}  Complete reference for your guidance: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html
Question: So, my files/folders structure is the following: project/ ├─ utils/ │ ├─ module.py ├─ server/ │ ├─ main.py  Inside project/server/main.py I'm trying to import project/utils/module.py using this syntax: from ..utils.module import my_function. I'm using VSCode, and it even autocomplete for me as I type the module path. But when I run the file project/server/main.py, I get the error in the title. I've read dozens of answers here on stack overflow about this topic but none of them used an example like this.

Answer: Here is a reference that explains this problem well. Basically, the problem is that __package__ is not set when running standalone scripts. File structure . └── project ├── server │ └── main.py └── utils └── module.py  project/server/main.py if __name__ == '__main__': print(__package__)  Output $ python3 project/server/main.py None  As we can see, the value of __package__ is None. This is a problem because it is the basis of relative imports as stated here:  __package__ ... This attribute is used instead of __name__ to calculate explicit relative imports for main modules, as defined in PEP 366...  Where PEP 366 explains this further:  The major proposed change is the introduction of a new module level attribute, __package__. When it is present, relative imports will be based on this attribute rather than the module __name__ attribute.  To resolve this, you can run it as a module via -m flag instead of a standalone script. Output $ python3 -m project.server.main # This can be <python3 -m project.server> if the file was named project/server/__main__.py project.server  project/server/main.py from ..utils.module import my_function if __name__ == '__main__': print(__package__) print("Main") my_function()  Output $ python3 -m project.server.main project.server Main My function  Now, __package__ is set, which means it can now resolve the explicit relative imports as documented above.
Question: I have a class with a method that is annotated with the lru_cache annotation: CACHE_SIZE=16384 class MyClass: [...] @lru_cache(maxsize=CACHE_SIZE) def _my_method(self, texts: Tuple[str]): <some heavy text processing> def cache_info(self): return self._my_method.cache_info()  After running for a while, I look at the cache statistics through the cache_info() method: c = MyClass() [...] c.cache_info() { "hits":9348, "misses":4312, "maxsize":16384, "currsize":2588 }  My question is: how can currsize be smaller than misses AND smaller than maxsize? My understanding was: for each miss, the result is added to the cache, hence increasing the current size. Only when the current size has reached the maximum size, cached results are removed. Since the maximum size is not reached here yet, each miss should be cached, so currsize should equal misses at this point. However, that does not seem to be the way this works.

Answer: If your program is either multi-threaded, or recursive - basically, any sort of condition where _my_method() might be called again while another call is partially completed - then it's possible to see the behavior you're experiencing. lru_cache() is thread-aware and uses the following set of steps for size-limited caching:  make a hash key out of the wrapped function's arguments lock the cache in a with block:  look up the key in the cache if the key is in the cache, return the cached value else, if the key isn't in the cache, increase misses by 1   call the wrapped function lock the cache again  if the result is in the cache now, return it if the result still isn't in the cache, add it, possibly removing older entries, etc. etc.    In other words, the cached value may have been added by another thread while the wrapped function was called, but it's still counted as a miss. If you had multiple calls to _my_method() that looked up the same missing key, causing misses to be incremented but then resulting in the key appearing in the cache by the time _my_method() completes, misses will be higher than currsize.