eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: My goal is to receive csv files in S3, convert them to avro, and validate them against the appropriate schema in AWS. I created a series of schemas in AWS Glue Registry based on the .avsc files I already had: { "namespace": "foo", "type": "record", "name": "bar.baz", "fields": [ { "name": "column1", "type": ["string", "null"] }, { "name": "column2", "type": ["string", "null"] }, { "name": "column3", "type": ["string", "null"] } ] }  But once I try and pull the schemas from Glue the API doesn't seem to provide definition details: glue = boto3.client('glue') glue.get_schema( SchemaId={ 'SchemaArn': schema['SchemaArn'] } )  returns: { 'Compatibility': 'BACKWARD', 'CreatedTime': '2021-08-11T21:09:15.312Z', 'DataFormat': 'AVRO', 'LatestSchemaVersion': 2, 'NextSchemaVersion': 3, 'RegistryArn': '[my-registry-arn]', 'RegistryName': '[my-registry-name]', 'ResponseMetadata': { 'HTTPHeaders': { 'connection': 'keep-alive', 'content-length': '854', 'content-type': 'application/x-amz-json-1.1', }, 'HTTPStatusCode': 200, 'RetryAttempts': 0, }, 'SchemaArn': '[my-schema-arn]', 'SchemaCheckpoint': 2, 'SchemaName': '[my-schema-name]', 'SchemaStatus': 'AVAILABLE', 'UpdatedTime': '2021-08-11T21:09:17.312Z', }  Is there a way to programmatically retrieve the Glue Schema Registry definitions for a schema? Or am I taking the wrong approach here with what I'm trying to do?

Answer: After some more digging I found the somewhat confusingly named get_schema_version() method that I had been overlooking which returns the SchemaDefinition: { 'SchemaVersionId': 'string', 'SchemaDefinition': 'string', 'DataFormat': 'AVRO'|'JSON', 'SchemaArn': 'string', 'VersionNumber': 123, 'Status': 'AVAILABLE'|'PENDING'|'FAILURE'|'DELETING', 'CreatedTime': 'string' }
Question: Is it possible to do this:  with ttk.Notebook widget?

Answer: Yes, it is possible. You have to bind B1-Motion to a function, then use notebook.index("@x,y") to get the index of the tab at mouse position. You can then make use of notebook.insert() to insert at a particular position. import tkinter as tk from tkinter import ttk def reorder(event): try: index = notebook.index(f"@{event.x},{event.y}") notebook.insert(index, child=notebook.select()) except tk.TclError: pass root = tk.Tk() root.geometry("500x500") notebook = ttk.Notebook(root) notebook.pack(fill="both", expand=True) notebook.bind("<B1-Motion>", reorder) frame1 = ttk.Frame(notebook) frame2 = ttk.Frame(notebook) frame1.pack(fill='both', expand=True) frame2.pack(fill='both', expand=True) notebook.add(frame1, text='Stackoverflow') notebook.add(frame2, text='Github') root.mainloop()  output:
Question: t1 = torch.tensor([1,2,3]) t2 = torch.tensor([4,5,6]) t3 = torch.tensor([7,8,9]) torch.stack((t1,t2,t3),dim=1)  When implementing the torch.stack(), I can't understand how stacking is done for different dim. Here stacking is done for columns but I can't understand the details as to how it is done. It becomes more complicated dealing with 2-d or 3-D tensors. tensor([[1, 4, 7], [2, 5, 8], [3, 6, 9]]) 

Answer: Imagine have n tensors. If we stay in 3D, those correspond to volumes, namely rectangular cuboids. Stacking corresponds to combining those n volumes on an additional dimension: here a 4th dimension is added to host the n 3D volumes. This operation is in clear contrast with concatenation, where the volumes would be combined on one of the existing dimensions. So concatenation of three-dimensional tensors would result in a 3D tensor. Here is a possible representation of the stacking operations for limited dimensions sizes (up to three-dimensional inputs):  Where you chose to perform the stacking defines along which new dimension the stack will take place. In the above examples, the newly created dimension is last, hence the idea of "added dimension" makes more sense. In the following visualization, we observe how tensors can be stacked on different axes. This in turn affects the resulting tensor shape  For the 1D case, for instance, it can also happen on the first axis, see below:  With code: >>> x_1d = list(torch.empty(3, 2)) # 3 lines >>> torch.stack(x_1d, 0).shape # axis=0 stacking torch.Size([3, 2]) >>> torch.stack(x_1d, 1).shape # axis=1 stacking torch.Size([2, 3])   Similarly for two-dimensional inputs:  With code: >>> x_2d = list(torch.empty(3, 2, 2)) # 3 2x2-squares >>> torch.stack(x_2d, 0).shape # axis=0 stacking torch.Size([3, 2, 2]) >>> torch.stack(x_2d, 1).shape # axis=1 stacking torch.Size([2, 3, 2]) >>> torch.stack(x_2d, 2).shape # axis=2 stacking torch.Size([2, 2, 3])    With this state of mind, you can intuitively extend the operation to n-dimensional tensors.
Question: I'd like to create an edge list with weights as an attribute (counts number of pair occurrences - e.g., how many months have the pair a-b been together in the same group). The dataframe contains a monthly snapshot of people in a particular team (there are no duplicates on the monthly groups)     monthyear name     jun2020 a   jun2020 b   jun2020 c   jul2020 a   jul2020 b   jul2020 d     The output should look like the following (it's non-directional so a-b pair is the same as b-a):     node1 node2 weight     a b 2   b c 1   a c 1   a d 1   b d 1     I managed to create a new dataframe with the names combinations using the following: df1 = pd.DataFrame(data=list(combinations(df['name'].unique().tolist(), 2)), columns=['node1', 'node2'])  Now I'm not sure how to iterate over this new dataframe to populate the weights. How can this be done?

Answer: Assuming that there are no duplicates within each monthyear group, you can get all 2-combinations of names within each group and then group by the node names to obtain the weight. from itertools import combinations def get_combinations(group): return pd.DataFrame([sorted(e) for e in list(combinations(group['name'].values, 2))], columns=['node1', 'node2']) df = df.groupby('monthyear').apply(get_combinations)  This will give you an intermediate result:  node1 node2 monthyear jul2020 0 a b 1 a d 2 b d jun2020 0 a b 1 a c 2 b c  Now, calculate the weight: df = df.groupby(['node1', 'node2']).size().to_frame('weight').reset_index()  Final result:  node1 node2 weight 0 a b 2 1 a c 1 2 a d 1 3 b c 1 4 b d 1
Question: I have the following code: class Test: def __init__(self, name): self.name = name def __enter__(self): print(f'entering {self.name}') def __exit__(self, exctype, excinst, exctb) -> bool: print(f'exiting {self.name}') return True with Test('first') as test: print(f'in {test.name}') test = Test('second') with test: print(f'in {test.name}')  Running it produces the following output: entering first exiting first entering second in second exiting second  But I expected it to produce: entering first in first exiting first entering second in second exiting second  Why isn't the code within my first example called?

Answer: The __enter__ method should return the context object. with ... as ... uses the return value of __enter__ to determine what object to give you. Since your __enter__ returns nothing, it implicitly returns None, so test is None. with Test('first') as test: print(f'in {test.name}') test = Test('second') with test: print(f'in {test.name}')  So test is none. Then test.name is an error. That error gets raised, so Test('first').__exit__ gets called. __exit__ returns True, which indicates that the error has been handled (essentially, that your __exit__ is acting like an except block), so the code continues after the first with block, since you told Python everything was fine. Consider def __enter__(self): print(f'entering {self.name}') return self  You might also consider not returning True from __exit__ unless you truly intend to unconditionally suppress all errors in the block (and fully understand the consequences of suppressing other programmers' errors, as well as KeyboardInterrupt, StopIteration, and various system signals)
Question: I'm refactoring a .NET application to airflow. This .NET application uses multiple threads to extract and process data from a mongoDB (Without multiple threads the process takes ~ 10hrs, with multi threads i can reduce this) . In each documment on mongoDB I have a key value namedprocess. This value is used to control which thread process the documment. I'm going to develop an Airflow DAG to optimize this process. My doubt is about performance and the best way to do this. My application should have multiple tasks (I will control the process variable in the input of the python method). Or should I use only 1 task and use Python MultiThreading inside this task? The image below illustrates my doubt. Multi Task X Single Task (Multi Threading) I know that using MultiTask I'm going to do more DB Reads (1 per task). Although, using Python Multi Threading I know I'll have to do a lot of control processing inside de task method. What is the best, fastest and optimized way to do this?

Answer: It really depends on the nature of your processing. Multi-threading in Python can be limiting because of GIL (Global Interpreter Lock) - there are some operations that require exclusive lock, and this limit the parallelism it can achieve. Especially if you mix CPU and I/O operations the effects might be that a lot of time is spent by threads waiting for the lock. But it really depends on what you do - you need to experiment to see if GIL affects your multithreading. Multiprocessing (which is used by Airflow for Local Executor) is better because each process runs effectively a separate Python interpreter. So each process has it's own GIL - at the expense of resources used (each process uses it's own memory, sockets and so on). Each task in Airlfow will run in a separate process. However Airflow offers a bit more - it also offers multi-machine. You can run separate workers With X processes on Y machines, effectively running up to X*Y processes at a time. Unfortunately, Airflow is (currently) not well suited to run dynamic number of parallel tasks of the same type. Specifically if you would like to split load to process to N pieces and run each piece in a separate task - this would only really work if N is constant and does not change over time for the same DAG (like if you know you have 10 machines, with 4 CPUs, you's typically want to run 10*4 = 40 tasks at a time, so you'd have to split your job into 40 tasks. And it cannot change dynamically between runs really - you'd have to write your DAG to run 40 parallel tasks every time it runs. Not sure if I helped, but there is no single "best optimised" answer - you need to experiment and check what works best for your case.
Question: I am trying to iterate over a dictionary that looks like this: account_data = {"a": "44196397", "b": "2545086098", "c": "210623431", "d": "1374059147440820231", "e": "972970759416111104", "f": "1060627757812641792", "g": "1368361032796700674", "h": "910899153772916736", "i": "887748030304329728", "j": "1381341090", "k": "2735504155", "l": "150324112", }  The goal is to use each ID to scrape some data, therefor I got a method that takes the corresponding userID and gets the data from it. At first I had a method for every ID in the dict, but now I want to change it so that I got one method which iterates over the dictionary, takes one ID at a time and makes the API request, if finished the next is made and so on. The problem is I can't iterate over the dictionary, I always just access the first one in here. I am relatively new to Python since I mainly used Java. Maybe dictionary is the wrong data structure for this task? Any help appreciated. Edit: This is my old code to iterate over the dictionary: def iterate_over_dict(): for key, value in account_data.items(): return value  I then continue with using the id in this function: def get_latest_data(): chosen_id = iterate_over_dict() print('id: ', chosen_id) # get my tweets tweets = get_tweets_from_user_id(chosen_id) # get tweet_id of latest tweet tweet_id = tweets.id.values[0] # get tweet_text of latest tweet tweets = tweets.text.values[0] # check if new tweet - if true -> check if contains data = check_for_new_tweet(tweet_id, tweets) if data is not None: print("_________") print('1 ', data)  But I always only use the first one. I think in Java it wouldn't be a problem for me since I can just use an index to iterate from 0 to n, but is there something similar for dictionaries? I also want to run the get_latest_data method every time a new ID is chosen from the dict

Answer: Use for loop for iteration. dict = {'a': 1, 'b': 2, 'c': 3} for key, value in dict.items(): print(key+" "+ str(value)) for key in dict: print(key+ " "+str(dict[key]))  The first one iterates over items and gives you keys and values. The second one iterates over keys and then it is accessing value from the dictionary using the key.
Question: Suppose the Python package mypackage is at a non-standard location on my machine and I am running Python code in the VSCode interactive window. If I type import mypackage it will not be found. This can be remedied by doing sys.path.append("/path/to/mypackage"). However, I would like to set things up so that within a given project each time I open the interactive window a set of paths, like /path/to/mypackage, have already been added to the search path. Is there a way to do this?

Answer: You can do this to modify the PYTHONPATH:  Add these in the settings.json file to Modify the PYTHONPATH in the terminal: "terminal.integrated.env.windows": { "PYTHONPATH": "xxx/site-packages" }  Create a .env file under your workspace, and add these settings in it to modify the PYTHONPATH for the extension and debugger: PYTHONPATH=xxx/site-packages   You can refer to [here][1] to understand the effects of these two configurations.
Question: When trying to run undetected-chromedriver I was running into the following error: urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate 

Answer: If you're using macOS go to Macintosh HD > Applications > Python3.9 folder (or whatever version of python you're using) > double click on "Install Certificates.command" file.
Question: I'm trying to import models from a folder in the parent directory. Im using sys.path.append(). My project structure: -Project  folder1  file1.py ...   folder2  file2.py ...    In file1.py file: sys.path.append('../Project') from Project.folder2 import file2  I then get a: ModuleNotFoundError: No module named Project  I know there are other ways but this seems like the simplest. I'm not sure if I need to put the absolute path to the Project folder, but I'm hoping not since I'll be running this Project on different computers (diff abs path).

Answer: 2 errors in your code:  The Project directory is not just 1-level up. From the point of view of file1.py, it is actually 2 levels up. See this:  $ cd .. (venv) nponcian 1$ tree . └── Project ├── folder1 │ └── file1.py └── folder2 └── file2.py (venv) nponcian 1$ cd Project/folder1/ (venv) nponcian folder1$ ls .. folder1 folder2 (venv) nponcian folder1$ ls ../.. Project   Even if the above works, adding a relative path as a string would literally append that string as its raw value. So if you add a print(sys.path), it would display something like this:  ['/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '../Project', '.']   It literally added '../Project', so if python started searching for the target modules in folder2, it wouldn't still find them because how would it know where exactly is '../Project' relative to.  Solution What you need to add is the absolute path. If your problem is it might change, it is fine because we don't need a fixed absolute path. We can get the absolute path through the location of the current file being executed e.g. file1.py and then extracting the parent directory needed. Thus, this will work regardless if the absolute paths change because the way we are getting it is always relative to file1.py. Try this: Project/folder1/file1.py from pathlib import Path import sys sys.path.append(str(Path(__file__).parent.parent.parent)) # 1. <.parent> contains this file1.py 2. <.parent.parent> contains folder1 3. <.parent.parent.parent> contains Project ... the rest of the file
Question: So I want to create a function that generates consecutive numbers from 'start' to 'end' as many as 'size'. For the iteration, it will be calculated inside the function. But I have problem to set default value of parameter 'end'. Before I explain further, here's the code: # Look at this ------------------------------- # || # \/ def consecutive_generator(size=20, start=0, end=(size+start)): i = start iteration = (end-start)/size arr = [] temp_size = 0 while temp_size < size: arr.append(i) i += iteration temp_size += 1 return arr # with default end, so the 'end' parameter will be 11 c1= consecutive_generator(10, start=1) print(c1) # with end set c2= consecutive_generator(10, end=20) print(c2)  As can be seen above (on the default value of the 'end' parameter), what I want to achieve is the 'end' parameter whose default value is 'start' + 'size' parameters (then the iteration will be 1) The output will definitely be an error. So how can i do this? (this is my first time asking on stackoverflow sorry if i made a mistake) (Closed)

Answer: This is a pretty standard pattern: def consecutive_generator(size=20, start=0, end=None): if end is None: end = size + start
Question: I have a pandas.DataFrame df and would like to add a new column col with one single value "hello". I would like this column to be of dtype category with the single category "hello". I can do the following. df["col"] = "hello" df["col"] = df["col"].astype("category")   Do I really need to write df["col"] three times in order to achieve this? After the first line I am worried that the intermediate dataframe df might take up a lot of space before the new column is converted to categorical. (The dataframe is rather large with millions of rows and the value "hello" is actually a much longer string.)  Are there any other straightforward, "short and snappy" ways of achieving this while avoiding the above issues? An alternative solution is df["col"] = pd.Categorical(itertools.repeat("hello", len(df)))  but it requires itertools and the use of len(df), and I am not sure how memory usage is under the hood.

Answer: We can explicitly build the Series of the correct size and type instead of implicitly doing so via __setitem__ then converting: df['col'] = pd.Series('hello', index=df.index, dtype='category')  Sample Program: import pandas as pd df = pd.DataFrame({'a': [1, 2, 3]}) df['col'] = pd.Series('hello', index=df.index, dtype='category') print(df) print(df.dtypes) print(df['col'].cat.categories)   a col 0 1 hello 1 2 hello 2 3 hello a int64 col category dtype: object Index(['hello'], dtype='object')
Question: I have a class that extends a base class. Upon instantiation, I want to check if the subclass has one of the classes implemented from its base, but I'm not sure the best way. hasattr(self, '[method]') returns the method from super if not implemented by the child, so I'm trying to tell the difference. Here is an example: class Base : def __init__ ( self,) : pass def fail (self,) : pass # Now create the subclass w/o .fail class Task ( Base ) : def __init__ ( self, ): print( hasattr( self, 'fail' ) ) # < returns True  When Task() is instantiated it prints True because Task inherits .fail from Base. But in this case, I want to know that Task Does Not implement .fail, so I want a False returned somehow. It seems like I'm looking for something like isimplemented( self, 'fail' ). What am I missing?

Answer: I'm not sure I understand correctly, but it sounds like you might be looking for Abstract Base Classes. (Documentation here, tutorial here.) If you specify an abstractmethod in a base class that inherits from abc.ABC, then attempting to instantiate a subclass will fail unless that subclass overrides the abstractmethod. from abc import ABC, abstractmethod class Base(ABC): @abstractmethod def fail(self): pass class Task(Base): pass class Task2(Base): def fail(self): pass # this raises an exception # `fail` method has not been overridden in the subclass. t1 = Task() # this succeeds # `fail` method has been overridden in the subclass. t2 = Task2()  If you want a check to happen at class definition time rather than instance instantiation time, another option is to write an __init_subclass__ method in your base class, which is called every time you subclass your base class or you subclass a class inheriting from your base class. (You don't have to raise an exception in __init_subclass__ — you could just add a fail_overriden boolean attribute to the class, or do anything you like really.) class Base: def fail(self): pass def __init_subclass__(cls, **kwargs): if cls.fail == Base.fail: raise TypeError( 'Subclasses of `Base` must override the `fail` method' ) super().__init_subclass__(**kwargs) # this class definition raises an exception # because `fail` has not been overridden class Task(Base): pass # this class definition works fine. class Task2(Base): def fail(self): pass  And if you just want each instance to tell you whether fail was overridden in their subclass, you can do this: class Base: def __init__(self): print(type(self).fail != Base.fail) def fail(self): pass class Task(Base): def __init__(self): super().__init__() class Task2(Base): def __init__(self): super().__init__() def fail(self): pass t1 = Task() # prints "True" t2 = Task2() # prints "False"
Question: I have a pandas.DataFrame of the form index df df1 0 0 111 1 1 111 2 2 111 3 3 111 4 0 111 5 2 111 6 3 111 7 0 111 8 2 111 9 3 111 10 0 111 11 1 111 12 2 111 13 3 111 14 0 111 15 1 111 16 2 111 17 3 111 18 1 111 19 2 111 20 3 111  I want to create a dataframe in which column df repeats 0,1,2,3. But there is something missing in the data. I'm trying to fill in the blanks with 0 by appending row values. Here is my expected result: index df df1 0 0 111 1 1 111 2 2 111 3 3 111 4 0 111 5 1 0 6 2 111 7 3 111 8 0 111 9 1 0 10 2 111 11 3 111 12 0 111 13 1 111 14 2 111 15 3 111 16 0 111 17 1 111 18 2 111 19 3 111 20 0 0 21 1 111 22 2 111 23 3 111  How can I achieve this? edit: What should I do if my input is as below? index df1 df2 0 0 111 1 1 111 2 2 111 3 3 111 4 0 111 5 3 111 6 1 111 7 2 111  Here is my expected result: index df1 df2 0 0 111 1 1 111 2 2 111 3 3 111 4 0 111 5 1 0 6 2 0 7 3 111 8 0 0 9 1 111 10 2 111 11 3 0 

Answer: Using @Mozway's idea, and combining with some helper functions from pyjanitor, the missing values can be made explicit, and later filled. Again, this is just another option : # pip install pyjanitor import pandas as pd import janitor as jn (df.assign(temp = df.df.diff().le(0).cumsum()) .complete('df', 'temp') # helper function .fillna(0) # relevant if you care about the order .sort_values('temp', kind='mergesort') # helper function .select_columns('df*') # or .drop(columns='temp') ) df df1 0 0 111.0 6 1 111.0 12 2 111.0 18 3 111.0 1 0 111.0 7 1 0.0 13 2 111.0 19 3 111.0 2 0 111.0 8 1 0.0 14 2 111.0 20 3 111.0 3 0 111.0 9 1 111.0 15 2 111.0 21 3 111.0 4 0 111.0 10 1 111.0 16 2 111.0 22 3 111.0 5 0 0.0 11 1 111.0 17 2 111.0 23 3 111.0