eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have an open source package with lots of classes over different submodules. All classes have methods fit and transform, and inherit fit_transform from sklearn. All classes have docstrings that follow numpydoc with subheadings Parameters, Attributes, Notes, See Also, and Methods, where I list fit, transform and fit_transform. I copy an example of a class: class DropFeatures(BaseEstimator, TransformerMixin): """ Some description. Parameters ---------- features_to_drop : str or list, default=None Variable(s) to be dropped from the dataframe Methods ------- fit transform fit_transform """ def __init__(self, features_to_drop: List[Union[str, int]]): some init parameters def fit(self, X: pd.DataFrame, y: pd.Series = None): """ This transformer does not learn any parameter. Verifies that the input X is a pandas dataframe, and that the variables to drop exist in the training dataframe. Parameters ---------- X : pandas dataframe of shape = [n_samples, n_features] The input dataframe y : pandas Series, default = None y is not needed for this transformer. You can pass y or None. Returns ------- self """ some functionality return self def transform(self, X: pd.DataFrame): """ Drop the variable or list of variables from the dataframe. Parameters ---------- X : pandas dataframe The input dataframe from which features will be dropped Returns ------- X_transformed : pandas dataframe, shape = [n_samples, n_features - len(features_to_drop)] The transformed dataframe with the remaining subset of variables. """ some more functionality return X  In the conf.py for Sphinx I include: extensions = [ "sphinx.ext.autodoc", # Core Sphinx library for auto html doc generation from docstrings "sphinx.ext.autosummary", # Create neat summary tables for modules/classes/methods etc "sphinx.ext.intersphinx", # Link to other project's documentation (see mapping below) "sphinx_autodoc_typehints", # Automatically document param types (less noise in class signature) "numpydoc", "sphinx.ext.linkcode", ] numpydoc_show_class_members = False # generate autosummary even if no references autosummary_generate = True autosummary_imported_members = True  When I build the documents using sphinx-build -b html docs build, the docs are built perfectly fine, but I get 3 warnings per class, one for each of the methods, that says: warning: autosummary: stub file not found for the methods of the class. check your autosummary_generate settings  I've exhausted all my searching resources, and I am ready to give up. Would someone know either how to prevent that warning or how to make sphinx not print it to the console? I attach a copy of the error and I can provide a link to the PR to the repo if needed 

Answer: Ok, after 3 days, I nailed it. The secret is add a short description to the methods in the docstrings after the heading "Methods" instead of leaving them empty as I did. So: class DropFeatures(BaseEstimator, TransformerMixin): Some description. Parameters ---------- features_to_drop : str or list, default=None Variable(s) to be dropped from the dataframe Methods ------- fit: some description transform: some description fit_transform: some description
Question: I have a python project with a basic setup that looks like this: imptest.py utils/something.py utils/other.py  Here's what's in the scripts: imptest.py #!./venv/bin/python import utils.something as something import utils.other as other def main(): """ Main function. """ something.do_something() other.do_other() if __name__ == "__main__": main()  something.py #import other def do_something(): print("I am doing something") def main(): """ Main function """ do_something() #other.do_other() if __name__ == "__main__": main()  other.py def do_other(): print("do other thing!") def main(): """ Main function """ do_other() if __name__ == "__main__": main()  imptest.py is the main file that runs and calls the utils functions occasionally for some things. And as you can see, I have commented out some lines in "something.py" where I am importing "other" module for testing. But when I want to test certain functions in something.py, I have to run the file something.py and uncomment the import line. This feels like a bit of a clunky way of doing this. If I leave the import other  uncommented and run imptest.py, I get this error: Traceback (most recent call last): File "imptest.py", line 5, in <module> import utils.something as something File "...../projects/imptest/utils/something.py", line 3, in <module> import other ModuleNotFoundError: No module named 'other'  What's a better way of doing this?

Answer: The problem here is the path, Consider this directory structure main - utils/something.py - utils/other.py imptest.py  When you try to import other using relative path in to something.py, then you would do something like from . import other. This would work when you execute $ python something.py but would fail when you run $ python imptest.py because in the second scenario it searches for main/other.py which doesn't exist. So inorder to fix this issue, I would suggest that you write unit tests for something.py & other.py and run them using $ python -m (mod) command. ( I highly recommend this approach ) But.... if you really what your existing code to work without much modification then you can add these 2 lines in something.py file ( this works, but I don't recommend this approach ) import sys, os sys.path.append(os.getcwd()) # Adding path to this module folder into sys path import utils.other as other def do_something(): print("I am doing something") def main(): """ Main function """ do_something() other.do_other() if __name__ == "__main__": main()  Here are some references to get better understanding:  Unit testing in python Absolute vs Relative Imports in python
Question: I have successfully built and started the docker container, it is running perfectly, but when I try to access it [End point url 0.0.0.0:6001] I am getting a "socket hang up" error GET http://0.0.0.0:6001/ Error: socket hang up Request Headers User-Agent: PostmanRuntime/7.26.8 Accept: */* Postman-Token: <token> Host: 0.0.0.0:6001 Accept-Encoding: gzip, deflate, br Connection: keep-alive  Earlier it was working fine but when I removed the containers and images and rebuild it then I started getting this error I am using Postman to make GET request and I also tried Web browser Can anyone tell me whats the problem --Update-- Docker File Creating containers # Create Virtual Network $ sudo docker network create network1 # Using custom network as there are multiple containers # which communicate with each other # Create Containers $ sudo docker build -t form_ocr:latest . $ sudo docker run -d -p 6001:5000 --net network1 --name form_ocr form_ocr  netstat command output $ netstat -nltp ... tcp6 0 0 :::6001 :::* LISTEN -  docker container inspect output $ sudo docker container inspect <container-id>  output docker ps output $ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 835e8cb11eee form_ocr "python3 app.py" 16 hours ago Up 40 seconds 0.0.0.0:6001->5000/tcp form_ocr 

Answer: Try localhost:6001 not internet address You can also try any of your system local ipaddress , you find ipaddress by typing ifconfig or ipconfig if you are in linux or windows respectively
Question: I installed the version chromedriver 88 as requested but my version chrome is 87.0.4280.88 that is the last version (outside beta) while I am also asked to download version 88 of chrome Here is the error : selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 88 Current browser version is 87.0.4280.88 with binary path C:\Program Files\Google\Chrome\Application\chrome.exe  How can I resolve this problem?

Answer: Your ChromeDriver version and your installed version of Chrome need to match up. You are using ChromeDriver for Chrome version 87. Keep both version same. Check your Chrome version (Help -> About) and then find the correct ChromeDriver release. You could instead use webdriver-manager which can handle this for you. Chrome is 87.0.4280.88 ChromeDriver Version 87 download from here https://chromedriver.storage.googleapis.com/index.html?path=87.0.4280.88/
Question: In my game, I have a list of tuples (x,y) : solution = [(36, 37), (36, 36), (36, 35), (37, 35), (38, 35), (38, 34), (38, 33), (38, 32)]  This list describes the movements the player should do to move from point (36, 37) to point (38, 32). I want to simplify this list to the following : opti = [(36, 37), (36, 35), (38, 35), (38, 32)]  This means I want to reduce any series of steps where x is fixed (or y is fixed) to only the first and the last step. I'm struggling to figure out an algorithm to do this. I've been trying for more than two hours and here is what I'm currently trying to work on: solution = [(36, 37), (36, 36), (36, 35), (37, 35), (38, 35), (38, 34), (38, 33), (38, 32)] opti = [solution[0]] for i in range(len(solution)): if opti[-1][0] == solution[i][0]: pass elif opti[-1][1] == solution[i][1]: pass else: opti.append(solution[i])  In the end opti is equal to [(36, 37), (37, 35), (38, 34)] which is not what I want.. Can someone point me to the right way to do this?

Answer: You can try this: compare the previous and next location with the current location when iterating over the list(solution) to check if all the points are in the same line, pass if they are in the same line else append to the final (opti) list. solution = [(36, 37), (36, 36), (36, 35), (37, 35), (38, 35), (38, 34), (38, 33), (38, 32)] opti = [solution[0]] for i in range(1, len(solution) -1 ): if solution[i-1][0] == solution[i][0] and solution[i][0] == solution[i+1][0]: pass elif solution[i-1][1] == solution[i][1] and solution[i][1] == solution[i+1][1]: pass else: opti.append(solution[i]) opti.append(solution[-1]) print(opti)  output: [(36, 37), (36, 35), (38, 35), (38, 32)]  I hope this helps, feel free to reach out in case of any doubt.
Question: I would like to let my users decide their email backend on their own. That is why I have created the email relevant keys (host, port, username...) on the users and now I try to work this (see below) backend into my Django project. Working with the docs and the source code, my first attempt was to extend the default EmailBackend by my custom "UserBackend" which overrides the __init__ function like this: class UserBackend(EmailBackend): def __init__(self, user_id, host=None, port=None, username=None ...): user = User.objects.get(id=user_id) super().init(host=user.email_host, port=user.email_port ...)  As this method is called (I tried to send_mail from the shell) it gets no user_id. How can I approach this differently or how would I extend my attempts to do this? I wouldn't want to rewrite Djangos mail system entirely, as it works in itself.

Answer: send_email has a parameter called connection (link to docs) which seems to fit perfectly. You can get a connection by calling get_connection (link to docs) with the user's parameters. connection = get_connection(host=user.email_host, port=user.email_port, ...) send_email(connection=connection, ...)  If you'd like to support multiple backend types, get_connection also supports it.
Question: I have DataFrame similat to this. How to add new column with names of rows that have same value in one of the column? For example: Have this:  name building a blue b white c blue d red e blue f red  How to get this?  name building in_building_with a blue [c, e] b white [] c blue [a, e] d red [f] e blue [a, c] f red [d] 

Answer: This is approach(worst) I can only think of : r = df.groupby('building')['name'].agg(dict) df['in_building_with'] = df.apply(lambda x: [r[x['building']][i] for i in (r[x['building']].keys()-[x.name])], axis=1)   df: name building in_building_with 0 a blue [c, e] 1 b white [] 2 c blue [a, e] 3 d red [f] 4 e blue [a, c] 5 f red [d]   Approach:  Make a dictionary which will give your indices where the building occurs.   building blue {0: 'a', 2: 'c', 4: 'e'} red {3: 'd', 5: 'f'} white {1: 'b'} dtype: object    subtract the index of the current building from the list since you are looking at the element other than it to get the indices of appearance.   r[x['building']].keys()-[x.name]    Get the values at those indices and make them into a list.
Question: In my Python 3 script, I am trying to make a combination of three numbers from three different lists based on inputs. If the lists are the same size, there is no issue with zip. However, I want to be able to input a single number for a specific list and the script to repeat that number until the longest list is finished. This can be done with zip_longest. However, with fillvalue it is not possible to have separate fill values for separate lists. Taking this simple script as an example: from itertools import zip_longest list1=[1] list2=[4, 5, 6, 7, 8, 9] list3=[2] for l1, l2, l3 in zip_longest(list1, list2, list3): print(l1, l2, l3)  This is the actual result: # 1 4 2 # None 5 None # None 6 None # None 7 None # None 8 None # None 9 None  And this would be the result that I want: # 1 4 2 # 1 5 2 # 1 6 2 # 1 7 2 # 1 8 2 # 1 9 2  I already managed to do this specific task by manually creating different for loops and asking if a list is a constant or not, but zip_longest is so close to exactly what I need that I wonder if I am missing something obvious.

Answer: You could make use of logical or operator to use the last element of the shorter lists: from itertools import zip_longest list1 = [1] list2 = ["a", "b", "c", "d", "e", "f"] list3 = [2] for l1, l2, l3 in zip_longest(list1, list2, list3): print(l1 or list1[-1], l2, l3 or list3[-1])  Out: 1 a 2 1 b 2 1 c 2 1 d 2 1 e 2 1 f 2
Question: I try to do very simple - update a value of a nested column;however, I cannot figure out how Environment:  Apache Spark 2.4.5 Databricks 6.4 Python 3.7  dataDF = [ (('Jon','','Smith'),'1580-01-06','M',3000) ] schema = StructType([ StructField('name', StructType([ StructField('firstname', StringType(), True), StructField('middlename', StringType(), True), StructField('lastname', StringType(), True) ])), StructField('dob', StringType(), True), StructField('gender', StringType(), True), StructField('gender', IntegerType(), True) ]) df = spark.createDataFrame(data = dataDF, schema = schema) df = df.withColumn("name.firstname", lit('John')) df.printSchema() df.show() #Results #I get a new column instead of update root |-- name: struct (nullable = true) | |-- firstname: string (nullable = true) | |-- middlename: string (nullable = true) | |-- lastname: string (nullable = true) |-- dob: string (nullable = true) |-- gender: string (nullable = true) |-- gender: integer (nullable = true) |-- name.firstname: string (nullable = false) +--------------+----------+------+------+--------------+ | name| dob|gender|gender|name.firstname| +--------------+----------+------+------+--------------+ |[Jon, , Smith]|1580-01-06| M| 3000| John| +--------------+----------+------+------+--------------+ 

Answer: Need to wrangle with the column a bit as below: import pyspark.sql.functions as F df2 = df.select('*', 'name.*') \ .withColumn('firstname', F.lit('newname')) \ .withColumn('name', F.struct(*[F.col(col) for col in df.select('name.*').columns])) \ .drop(*df.select('name.*').columns) df2.show() +------------------+----------+------+------+ | name| dob|gender|gender| +------------------+----------+------+------+ |[newname, , Smith]|1580-01-06| M| 3000| +------------------+----------+------+------+
Question: Can Python delete specific multiple files in S3? I want to delete multiple files with specific extensions. This script removes all files. These are the various specific files that I want to delete: XXX.tar.gz XXX.txt  ** Current code: ** (all files deleted) import boto3 accesskey = "123" secretkey = "123" region = "ap-northeast-1" s3 = boto3.resource ('s3', aws_access_key_id = accesskey, aws_secret_access_key = secretkey, region_name = region) bucket = s3.Bucket ('test') files = [os.key for os in bucket.objects.filter (Prefix = "myfolder / test /")] tar_files = [file to file in files if file.endswith ('tar.gz')] #print (f'All files: {files} ') #print (f'CSV files: {csv_files} ') objects_to_delete = s3.meta.client.list_objects (Bucket = "test", Prefix = "myfolder / test /") delete_keys = {'Objects': []} delete_keys ['Objects'] = [{'Key': tar_files} for tar_files in [obj ['Key'] for obj in objects_to_delete.get ('Content', [])]] s3.meta.client.delete_objects (Bucket = "test", Delete = delete_keys)  If anyone knows, please let me know.

Answer: Presuming that you want to delete *.tar.gz and *.txt files from the given bucket and prefix, this would work: import boto3 s3_resource = boto3.resource('s3') bucket = s3_resource.Bucket('my-bucket') objects = bucket.objects.filter(Prefix = 'myfolder/') objects_to_delete = [{'Key': o.key} for o in objects if o.key.endswith('.tar.gz') or o.key.endswith('.txt')] if len(objects_to_delete): s3_resource.meta.client.delete_objects(Bucket='my-bucket', Delete={'Objects': objects_to_delete})
Question: Assume we have a numpy array A with shape (N, ) and a matrix D with shape (M, 3) which has data and another matrix I with shape (M, 3) which has corresponding index of each data element in D. How can we construct A given D and I such that the repeated element indexes are added? Example: ############# A[I] := D ################################### A = [0.5, 0.6] # Final Reduced Data Vector D = [[0.1, 0.1 0.2], [0.2, 0.4, 0.1]] # Data I = [[0, 1, 0], [0, 1, 1]] # Indices  For example: A[0] = D[0][0] + D[0][2] + D[1][0] # 0.5 = 0.1 + 0.2 + 0.2  Since in index matrix we have: I[0][0] = I[0][2] = I[1][0] = 0  Target is to avoid looping over all elements to be efficient for large N, M (10^6-10^9).

Answer: I doubt you can get much faster than np.bincount - and notice how the official documentation provides this exact usecase # Your example A = [0.5, 0.6] D = [[0.1, 0.1, 0.2], [0.2, 0.4, 0.1]] I = [[0, 1, 0], [0, 1, 1]] # Solution import numpy as np D, I = np.array(D).flatten(), np.array(I).flatten() print(np.bincount(I, D)) #[0.5 0.6]
Question: With Chrome you can add options when creating the driver. You just do options = Options() options.headless = True driver = webdriver.Chrome(PATH\TO\DRIVER, options=options)  But for some reason when trying to do the same with Microsoft Edge options = Options() options.headless = True driver = webdriver.Edge(PATH\TO\DRIVER, options=options)  I get this error below:  TypeError: init() got an unexpected keyword argument 'options'  For some reason Edge's driver doesn't accept any other parameters than the file path. Is there any way to run Edge headless and add more options just like in Chrome?

Answer:  options = EdgeOptions() options.use_chromium = True options.add_argument("headless") options.add_argument("disable-gpu")  Try above code , you have to enable chromium to enable headless https://learn.microsoft.com/en-us/microsoft-edge/webdriver-chromium/?tabs=python This works only for new edge chromium not for edge legacy versions . In legacy versions headless is not supported Full code from msedge.selenium_tools import EdgeOptions from msedge.selenium_tools import Edge # make Edge headless edge_options = EdgeOptions() edge_options.use_chromium = True # if we miss this line, we can't make Edge headless # A little different from Chrome cause we don't need two lines before 'headless' and 'disable-gpu' edge_options.add_argument('headless') edge_options.add_argument('disable-gpu') driver = Edge(executable_path='youredgedriverpath', options=edge_options)
Question: Python (3.8) code: #!/usr/bin/env python3 from decimal import Decimal from decimal import getcontext x = Decimal('0.6666666666666666666666666667') y = x; print(getcontext().prec) print(y) print(y == x) y += x; y += x; y += x; y -= x; y -= x; y -= x; print(y) print(y == x)  Python output: 28 0.6666666666666666666666666667 True 0.6666666666666666666666666663 False   Java code: import java.math.BigDecimal; public class A { public static void main(String[] args) { BigDecimal x = new BigDecimal("0.6666666666666666666666666667"); BigDecimal y = new BigDecimal("0.6666666666666666666666666667"); System.out.println(x.precision()); System.out.println(y.precision()); System.out.println(y); System.out.println(y.equals(x)); y = y.add(x); y = y.add(x); y = y.add(x); y = y.subtract(x); y = y.subtract(x); y = y.subtract(x); System.out.println(y); System.out.println(y.equals(x)); } }  Java output: 28 28 0.6666666666666666666666666667 true 0.6666666666666666666666666667 true  What would be the way to achieve arbitrary precision in Python? By setting a very large prec?

Answer: From Python documentation:  The decimal module incorporates a notion of significant places so that 1.30 + 1.20 is 2.50.  Moreover, the following also need to be considered:  The context precision does not affect how many digits are stored. That is determined exclusively by the number of digits in value. For example, Decimal('3.00000') records all five zeros even if the context precision is only three.   Context precision and rounding only come into play during arithmetic operations.  Therefore: import decimal from decimal import Decimal decimal.getcontext().prec = 4 a = Decimal('1.22222') #1.22222 #what you put in is what you get even though the prec was set to 4 print(a) b = Decimal('0.22222') #0.22222 #Same reasoning as above print(b) a += 0; b += 0 #a will be printed as 1.222 (4 significant figures) #b will be printed as 0.2222 (Leading zeroes are not significant!) print('\n', a, '\n', b, sep='')
Question: I would like to know if the spacy tokenizer could tokenize words only using the "space" rule. For example: sentence= "(c/o Oxford University )"  Normally, using the following configuration of spacy: nlp = spacy.load("en_core_news_sm") doc = nlp(sentence) for token in doc: print(token)  the result would be:  ( c / o Oxford University )  Instead, I would like an output like the following (using spacy): (c/o Oxford University )  Is it possible to obtain a result like this using spacy?

Answer: Let's change nlp.tokenizer with a custom Tokenizer with token_match regex: import re import spacy from spacy.tokenizer import Tokenizer nlp = spacy.load('en_core_web_sm') text = "This is it's" print("Before:", [tok for tok in nlp(text)]) nlp.tokenizer = Tokenizer(nlp.vocab, token_match=re.compile(r'\S+').match) print("After :", [tok for tok in nlp(text)])   Before: [This, is, it, 's] After : [This, is, it's]  You can further adjust Tokenizer by adding custom suffix, prefix, and infix rules. An alternative, more fine grained way would be to find out why it's token is split like it is with nlp.tokenizer.explain(): import spacy from spacy.tokenizer import Tokenizer nlp = spacy.load('en_core_web_sm') text = "This is it's. I'm fine" nlp.tokenizer.explain(text)   You'll find out that split is due to SPECIAL rules: [('TOKEN', 'This'), ('TOKEN', 'is'), ('SPECIAL-1', 'it'), ('SPECIAL-2', "'s"), ('SUFFIX', '.'), ('SPECIAL-1', 'I'), ('SPECIAL-2', "'m"), ('TOKEN', 'fine')]  that could be updated to remove "it's" from exceptions like: exceptions = nlp.Defaults.tokenizer_exceptions filtered_exceptions = {k:v for k,v in exceptions.items() if k!="it's"} nlp.tokenizer = Tokenizer(nlp.vocab, rules = filtered_exceptions) [tok for tok in nlp(text)]   [This, is, it's., I, 'm, fine]  or remove split on apostrophe altogether: filtered_exceptions = {k:v for k,v in exceptions.items() if "'" not in k} nlp.tokenizer = Tokenizer(nlp.vocab, rules = filtered_exceptions) [tok for tok in nlp(text)]   [This, is, it's., I'm, fine]  Note the dot attached to the token, which is due to the suffix rules not specified.