eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: How do I sum duplicate elements in a list of lists of dictionaries? Sample list: data = [ [ {'user': 1, 'rating': 0}, {'user': 2, 'rating': 10}, {'user': 1, 'rating': 20}, {'user': 3, 'rating': 10} ], [ {'user': 4, 'rating': 4}, {'user': 2, 'rating': 80}, {'user': 1, 'rating': 20}, {'user': 1, 'rating': 10} ], ]  Expected output: op = [ [ {'user': 1, 'rating': 20}, {'user': 2, 'rating': 10}, {'user': 3, 'rating': 10} ], [ {'user': 4, 'rating': 4}, {'user': 2, 'rating': 80}, {'user': 1, 'rating': 30}, ], ] 

Answer: You can try: from itertools import groupby result = [] for lst in data: sublist = sorted(lst, key=lambda d: d['user']) grouped = groupby(sublist, key=lambda d: d['user']) result.append([ {'user': name, 'rating': sum([d['rating'] for d in group])} for name, group in grouped]) # Sort the `result` `rating` wise: result = [sorted(sub, key=lambda d: d['rating']) for sub in result] # %%timeit # 7.54 µs ± 220 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)   UPDATE (A more efficient solution): result = [] for lst in data: visited = {} for d in lst: if d['user'] in visited: visited[d['user']]['rating'] += d['rating'] else: visited[d['user']] = d result.append(sorted(visited.values(), key=lambda d: d['rating'])) # %% timeit # 2.5 µs ± 54 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  Result: # print(result) [ [ {'user': 2, 'rating': 10}, {'user': 3, 'rating': 10}, {'user': 1, 'rating': 20} ], [ {'user': 4, 'rating': 4}, {'user': 1, 'rating': 30}, {'user': 2, 'rating': 80} ] ]
Question: I have two apps: collection and accounts, with both having models defined. I'm importing a model ReporterProfile from accounts to collection. Similarly, I'm importing a model Report from collection to accounts. The Report model from collection is called in a model class method in accounts like this: from collection.models import Report class ReporterProfile(models.Model): .... def published_articles_number(self): num = Report.objects.filter(reporterprofile=self.id).count() return num  Similarly, I am importing ReporterProfile and User models from accounts to collection model like this: from accounts.models import ReporterProfile, User from <project_name> import settings class Report(models.Model): reporterprofile = models.ForeignKey(ReporterProfile, on_delete=models.CASCADE, verbose_name="Report Author") ... class Comment(models.Model): report = models.ForeignKey(Report, on_delete=models.CASCADE, related_name='comments') user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, verbose_name="Comment by") ...  When running the server or makemigrations, I get the error: File "F:\project_name\accounts\models.py", line 8, in <module> from collection.models import Report File "F:\project_name\collection\models.py", line 2, in <module> from accounts.models import ReporterProfile, User ImportError: cannot import name 'ReporterProfile' from partially initialized module 'accounts.models' (most likely due to a circular import) (F:\project_name\accounts\models.py)  I think the error is coming because of a wrong importing pattern. What should I do?

Answer: For ForeignKey: Instead of using reporterprofile = models.ForeignKey(ReporterProfile, ...), you can use reporterprofile = models.ForeignKey("accounts.ReporterProfile", ...), so you don't have to import the model. For preventing circulor import error : Instead of using : from accounts.models import ReporterProfile [...] foo = ReporterProfile()  You can use: import accounts.models [...] foo = accounts.models.ReporterProfile()
Question: I have the following example df:  col1 col2 col3 doc_no 0 a x f 0 1 a x f 1 2 b x g 2 3 b y g 3 4 c x t 3 5 c y t 4 6 a x f 5 7 d x t 5 8 d x t 6  I want to group by the first 3 columns (col1, col2, col3), concatenate the fourth column (doc_no) into a line of strings based on the groupings of the first 3 columns, as well as also generate a sorted count column of the 3 column grouping (count). Example desired output below (column order doesn't matter):  col1 col2 col3 count doc_no 0 a x f 3 0, 1, 5 1 d x t 2 5, 6 2 b x g 1 2 3 b y g 1 3 4 c x t 1 3 5 c y t 1 4  How would I go about doing this? I used the below line to get just the grouping and the count: grouped_df = df.groupby(['col1','col2','col3']).size().reset_index(name='count')\ .sort_values(['count'], ascending=False).reset_index()  But I'm not sure how to also get the concatenated doc_no column in the same code line.

Answer: Try groupby and agg like so: (df.groupby(['col1', 'col2', 'col3'])['doc_no'] .agg(['count', ('doc_no', lambda x: ','.join(map(str, x)))]) .sort_values('count', ascending=False) .reset_index()) col1 col2 col3 count doc_no 0 a x f 3 0,1,5 1 d x t 2 5,6 2 b x g 1 2 3 b y g 1 3 4 c x t 1 3 5 c y t 1 4  agg is simple to use because you can specify a list of reducers to run on a single column.
Question: Imagine one has two classes derived from Enum, e.g. class Color(Enum): blue = 'blue' red = 'red' class Properties(Enum): height = 'h' weight = 'w' colors = Color  What is the best way to (probably recursively) iterate over all Enum-labels of a nested Enum like Properties, including the ones of Enum-members like Properties.colors in the example above (i.e. including Color.blue and Color.red)? Checking for the type of the value?

Answer: Here's a quick example that just prints them out. I'll leave it as an exercise to the reader to make this a generic generator or whatever applies to the actual use case. :) >>> from typing import Type >>> def print_enum(e: Type[Enum]) -> None: ... for p in e: ... try: ... assert(issubclass(p.value, Enum)) ... print_enum(p.value) ... except (AssertionError, TypeError): ... print(p) ... >>> print_enum(Properties) Properties.height Properties.weight Color.blue Color.red
Question: I have the following when I print my data structure: print(speed_tf) 44.0 -24.4 45.0 -12.2 46.0 -12.2 47.0 -12.2 48.0 -12.2 Name: Speed, dtype: float64  I believe this is a pandas Series but not sure I do not want the first column at all I just want  -24.4 -12.2 -12.2 -12.2 -12.2  I tried speed_tf.reset_index()  index Speed 0 44.0 -24.4 1 45.0 -12.2 2 46.0 -12.2 3 47.0 -12.2 4 48.0 -12.2  How can I just get the Speed values with index starting at 0?

Answer: speed_tf.values  Should do what you want.
Question: I am looking for a perfect way to smooth edges of binary images. The problem is the binary image appears to be a staircase like borders which is very unpleasing for my further masking process.  I am attaching a raw binary image that is to be converted into smooth edges and I am also providing the expected outcome. I am also looking for a solution that would work even if we increase the dimensions of the image. Problem Image Expected Outcome

Answer: You can do that in Python/OpenCV with the help of Skimage by blurring the binary image. Then apply a one-sided clip. Input:  import cv2 import numpy as np import skimage.exposure # load image img = cv2.imread('bw_image.png') # blur threshold image blur = cv2.GaussianBlur(img, (0,0), sigmaX=3, sigmaY=3, borderType = cv2.BORDER_DEFAULT) # stretch so that 255 -> 255 and 127.5 -> 0 # C = A*X+B # 255 = A*255+B # 0 = A*127.5+B # Thus A=2 and B=-127.5 #aa = a*2.0-255.0 does not work correctly, so use skimage result = skimage.exposure.rescale_intensity(blur, in_range=(127.5,255), out_range=(0,255)) # save output cv2.imwrite('bw_image_antialiased.png', result) # Display various images to see the steps cv2.imshow('result', result) cv2.waitKey(0) cv2.destroyAllWindows()    You will have to adjust the amount of blur for the degree of aliasing in the image.
Question: I'm building a simple Flask based app using Cloud Run + Cloud Firestore. There is one method that brings a lot of data, and the logs are showing this error: `Memory limit of 244M exceeded with 248M used. Consider increasing the memory limit, see https://cloud.google.com/run/docs/configuring/memory-limits`  How I can increase the memory limit in the cloudbuild.yaml? Our YAML file contains the following: # cloudbuild.yaml steps: # build & push the container image - name: "gcr.io/kaniko-project/executor:latest" args: ["--cache=true", "--cache-ttl=48h", "--destination=gcr.io/$PROJECT_ID/todo:latest"] # Deploy container image to Cloud Run - name: "gcr.io/cloud-builders/gcloud" args: ['beta', 'run', 'deploy', 'todo', '--image', 'gcr.io/$PROJECT_ID/todo:latest', '--region', 'us-central1', '--allow-unauthenticated', '--platform', 'managed']  Thank you

Answer: In the args of the last step, add '--memory', '512Mi' The format for size is a fixed or floating point number followed by a unit: G, M, or K corresponding to gigabyte, megabyte, or kilobyte, respectively, or use the power-of-two equivalents: Gi, Mi, Ki corresponding to gibibyte, mebibyte or kibibyte respectively.
Question: I am using tensorflow 2.1 along with python 3.7 The following snippet of code is being used to build a tensorflow graph. The code runs without errors when executed as a standalone python script. (Probably tensorflow is running in eager mode? I am not sure.)  import tensorflow as tf patches = tf.random.uniform(shape=(1, 10, 50, 300), dtype=tf.dtypes.float32) s = tf.shape(patches) patches = [patches[0][x][y] - tf.reduce_mean(patches[0][x][y]) for y in tf.range(s[2]) for x in tf.range(s[1])]  However, the code fails when this is part of a tensorflow graph. I receive the following error: tensorflow.  python.framework.errors_impl.OperatorNotAllowedInGraphError: iterating over tf.Tensor is not allowed: AutoGraph is disabled in this function. Try decorating it directly with @tf.function.  I also added the decorator @tf.function to the method which wraps the above lines of code. It didn't help. I am not sure if I fully understand the meaning of decorating with @tf.function. I also checked that this could be a problem with using python list comprehension inside the tensorflow graph. I am not sure how to use tf.map_fn or tf.while_loop for my case, since I have nested loops.  Thanks in advance!

Answer: List comprehensions are not yet supported in autograph. The error that's raised needs to be improved, too. Piling up on https://github.com/tensorflow/tensorflow/issues/32546 should help resolve it sooner. Until comprehensions are supported, you have to use map_fn, which in this case would look something like this: def outer_comp(x): def inner_comp(y): return patches[0][x][y] - tf.reduce_mean(patches[0][x][y]) return tf.map_fn(inner_comp, tf.range(s[2]), dtype=tf.float32) patches = tf.map_fn(outer_comp, tf.range(s[1]), dtype=tf.float32)  That said, I believe you can just use reduce_mean directly: patches = patches - tf.expand_dims(tf.reduce_mean(patches, axis=3), -1)
Question: I'm using Windows 10 and Python 3.7. I ran the following command. import subprocess exeFilePath = "C:/Users/test/test.exe" subprocess.run(exeFilePath)  The .exe file launched with this command, I want to force-quit when the button is clicked or when the function is executed. Looking at a past question, it has been indicated that the way to force quit is to get a PID and do an OS.kill as follows. import signal os.kill(self.p.pid, signal.CTRL_C_EVENT)  However, I don't know how to get the PID of the process started in subprocess.run. What should I do?

Answer: Assign a variable to your subprocess import os import signal import subprocess exeFilePath = "C:/Users/test/test.exe" p = subprocess.Popen(exeFilePath) print(p.pid) # the pid os.kill(p.pid, signal.SIGTERM) #or signal.SIGKILL  In same cases the process has children processes. You need to kill all processes to terminate it. In that case you can use psutil  #python -m pip install —user psutil import psutil #remember to assign subprocess to a variable def kills(pid): '''Kills all process''' parent = psutil.Process(pid) for child in parent.children(recursive=True): child.kill() parent.kill() #assumes variable p kills(p.pid)  This will kill all processes in that PID
Question: I often use GridSearchCV for hyperparameter tuning. For example, for tuning regularization parameter C in Logistic Regression. Whenever an estimator I am using has its own n_jobs parameter I am confused where to set it, in estimator or in GridSearchCV, or in both? Same thing applies to cross_validate.

Answer: This is a very interesting question. I don't have a definitive answer, but some elements that are worth mentioning to understand the issue, and don't fir in a comment. Let's start with why you should or should not use multiprocessing :   Multiprocessing is useful for independent tasks. This is the case in a GridSearch, where all your different variations of your models are independent. Multiprocessing is not useful / make things slower when :  Task are too small : creating a new process takes time, and if your task is really small, this overhead with slow the execution of the whole code Too many processes are spawned : your computer have a limited number of cores. If you have more processes than cores, a load balancing mechanism will force the computer to regularly switch the processes that are running. These switches take some time, resulting in a slower execution.   The first take-out is that you should not use n_jobs in both GridSearch and the model you're optimizing, because you will spawn a lot of processes and end up slowing the execution. Now, a lot of sklearn models and functions are based on Numpy/SciPy which in turn, are usually implemented in C/Fortran, and thus already use multiprocessing. That means that these should not be used with n_jobs>1 set in the GridSearch. If you assume your model is not already parallelized, you can choose to set n_jobsat the model level or at the GridSearch level. A few models are able to be fully parallelized (RandomForest for instance), but most may have at least some part that is sequential (Boosting for instance). In the other end, GridSearch has no sequential component by design, so it would make sense to set n_jobs in GridSearch rather than in the model. That being said, it depend on the implementation of the model, and you can't have a definitive answer without testing for yourself for your case. For example, if you pipeline consume a lot of memory for some reason, setting n_jobs in the GridSearch may cause memory issues. As a complement, here is a very interesting note on parallelism in sklearn
Question: I've tried to load pre-trained FastText vectors from fastext - wiki word vectors. My code is below, and it works well.  from gensim.models import FastText model = FastText.load_fasttext_format('./wiki.en/wiki.en.bin')  but, the warning message is a little annoying.  gensim_fasttext_pretrained_vector.py:13: DeprecationWarning: Call to deprecated `load_fasttext_format` (use load_facebook_vectors (to use pretrained embeddings)  The message said, load_fasttext_format will be deprecated so, it will be better to use load_facebook_vectors.  So I decided to changed the code. and My changed code is like below. from gensim.models import FastText model = FastText.load_facebook_vectors('./wiki.en/wiki.en.bin')  But, the error occurred, the error message is like this.  Traceback (most recent call last): File "gensim_fasttext_pretrained_vector.py", line 13, in <module> model = FastText.load_facebook_vectors('./wiki.en/wiki.en.bin') AttributeError: type object 'FastText' has no attribute 'load_facebook_vectors'  I couldn't understand why these thing happen. I just change what the messages said, but it doesn't work. If you know anything about this, please let me know.  Always, thanks for you guys help. 

Answer: You're almost there, you need to change two things:  First of all, it's fasttext all lowercase letters, not Fasttext. Second of all, to use load_facebook_vectors, you need first to create a datapath object before using it.  So, you should do like so: from gensim.models import fasttext from gensim.test.utils import datapath wv = fasttext.load_facebook_vectors(datapath("./wiki.en/wiki.en.bin"))
Question: I want to use Conda to create a virtual environment from a YAML file. However, many packages end up with a Conflict error. The best way to solve this is to install each package individually instead of creating a virtual environment from a YAML file, right? If anyone knows of a better way to do it, please let me know.

Answer: Use conda-forge which has a strong dependency resolution implementation. Newer conda versions (>=4.6) introduced a strict channel priority feature. Type conda config --describe channel_priority  for more information. The solution is to add the conda-forge channel on top of defaults in your .condarc file when using conda-forge packages and activate the strict channel priority with: $ conda config --set channel_priority strict  This will ensure that all the dependencies will come from the conda-forge channel unless they exist only on defaults. You could also use Pipenv, and the Pipfile feature it comes with. Pipenv will attempt to install sub-dependencies that satisfy all the requirements from your core dependencies. see more: https://realpython.com/pipenv-guide/
Question: I have some cropped images and I need images that have black texts on white background. Firstly I apply adaptive thresholding and then I try to remove noise. Although I tried a lot of noise removal techniques but when the image changed, the techniques I used failed.  The best method for converting image color to binary for my images is Adaptive Gaussian Thresholding. Here is my code: im_gray = cv2.imread("image.jpg", cv2.IMREAD_GRAYSCALE) image = cv2.GaussianBlur(im_gray, (5,5), 1) th = cv2.adaptiveThreshold(image,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY,3,2)   I need smooth values, Decimal separator(dot) and postfix letters. How can I do this?

Answer: Before binarization, it is necessary to correct the nonuniform illumination of the background. For example, like this: import cv2 image = cv2.imread('9qBsB.jpg') image=cv2.cvtColor(image,cv2.COLOR_BGR2GRAY) se=cv2.getStructuringElement(cv2.MORPH_RECT , (8,8)) bg=cv2.morphologyEx(image, cv2.MORPH_DILATE, se) out_gray=cv2.divide(image, bg, scale=255) out_binary=cv2.threshold(out_gray, 0, 255, cv2.THRESH_OTSU )[1] cv2.imshow('binary', out_binary) cv2.imwrite('binary.png',out_binary) cv2.imshow('gray', out_gray) cv2.imwrite('gray.png',out_gray)  Result:
Question: How can I archive this? from tqdm import tqdm for link in tqdm(links): try: #Do Some Stff except: pass print("Done:")  Result:  100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 111.50it/s] Done: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 111.50it/s] Done:  Expected Result (Showing the status bar but don't print it after into the console) Done: Done: 

Answer: tqdm actually takes several arguments, one of them is leave, which according to the docs:  If [default: True], keeps all traces of the progressbar upon termination of iteration. If None, will leave only if position is 0  So: >>> for _ in tqdm(range(2)): ... time.sleep(1) ... 100%|██████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.01s/it]  Whereas setting leave=False yields: >>> for _ in tqdm(range(2), leave=False): ... time.sleep(1) ... >>>
Question: I have an Excel sheet with 100 rows. Each one has various informations, including an id, and a cell containing a photo. I use pandas to load the data into dictionaries : import pandas as pd df = pd.read_excel('myfile.xlsx') data = [] for index,row in df.iterrows(): data.append({ 'id':row['id'], 'field2':row['field2'], 'field3':row['field3'] })  For the image column, I want to extract each image, name it with the id of the row (image_row['id'].jpg) and put it into a folder. Then, I want to store the path to the image as below : for index,row in df.iterrows(): data.append({ 'id':row['id'], 'field2':row['field2'], 'field3':row['field3'], 'image':'path/image_'+row['id']+'.jpg' })  I'm looking for a way to do that, or another way if better. Do you have any idea ? I'm on Linux, so I can't use this method with pywin32. Thanks a lot -- EDIT You can find here an exemple of sheet i use

Answer: I found a solution using openpyxl and openpyxl-image-loader modules # installing the modules pip3 install openpyxl pip3 install openpyxl-image-loader  Then, in the script : #Importing the modules import openpyxl from openpyxl_image_loader import SheetImageLoader #loading the Excel File and the sheet pxl_doc = openpyxl.load_workbook('myfile.xlsx') sheet = pxl_doc['Sheet_name'] #calling the image_loader image_loader = SheetImageLoader(sheet) #get the image (put the cell you need instead of 'A1') image = image_loader.get('A1') #showing the image image.show() #saving the image image.save('my_path/image_name.jpg')  In the end, I can store the path and the image name in my dictionaries in a loop for each row
Question: Is there any obvious way to validate a pydantic model after changing some attribute? Say I create a simple Model and object: from pydantic import BaseModel class A(BaseModel): b: int = 0 a=A()  Then edit it, so that it is actually invalid: a.b = "foobar"  Can I force a re-validation and expect a ValidationError to be raised? I tried A.validate(a) # no error a.copy(update=dict(b='foobar')) # no error  What did work was A(**dict(a._iter())) ValidationError: 1 validation error for A b value is not a valid integer (type=type_error.integer)  But that is not really straightforward and I need to use the supposedly private method _iter. Is there a clean alternative?

Answer: pydantic can do this for you, you just need validate_assignment: from pydantic import BaseModel class A(BaseModel): b: int = 0 class Config: validate_assignment = True
Question: I am trying to install mininet-wifi. After downloading it, I have been using the following command to install it:  sudo util/install.sh -Wlnfv  However, I keep getting the error:  E: Unable to locate package python-pip  I have tried multiple times to download python-pip. I know mininet-wifi utilizes python 2 instead of python 3. I have tried to download python-pip using the command:  sudo apt-get install python-pip  But that leads to the same error:  E: Unable to locate package python-pip 

Answer: Pip for Python 2 is not included in the Ubuntu 20.04 repositories.  You need to install pip for Python 2 using the get-pip.py script.  1. Start by enabling the universe repository: sudo add-apt-repository universe  2. Update the packages index and install Python 2: sudo apt update sudo apt install python2  3. Use curl to download the get-pip.py script specific to python 2.7: curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py  4. Once the repository is enabled, run the script as sudo user with python2 to install pip : sudo python2 get-pip.py    Pip will be installed globally. If you want to install it only for your user, run the command without sudo. The script will also install setuptools and wheel, which allow you to install source distributions Verify the installation by printing the pip version number: pip2 --version  The output will look something like this:  pip 20.0.2 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)