eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a histogram of an image in RGB which represents the three curves of the three components R, G and B. I want to find the inflection points of each curve. I used the second derivative to find them but I can't, the second derivative does not cancel its returns null. So how can I find the inflection point? Is there any other method to find them?  import os, cv2, random import numpy as np import matplotlib.pyplot as plt import math from sympy import * image = cv2.imread('C:/Users/Xers/Desktop/img.jpg') CHANNELS = ['r', 'g', 'b'] for i, channel in enumerate( CHANNELS ): histogram = cv2.calcHist([image], [i], None, [256], [0,256]) histogram = cv2.GaussianBlur( histogram, (5,5), 0) plt.plot(histogram, color = channel) x= plt.xlim([0,256]) y = plt.ylim([0, 24000]) derivative1= np.diff(histogram, axis=0) derivative2= np.diff(derivative1, axis=0) inf_point = np.where ( derivative2 == 0)[0] print(inf_point) plt.show() 

Answer: There are two issues of numerical nature with your code:  the data does not seem to be continuous enough to rely on the second derivative computed from two subsequent np.diff() applications even if it were, the chances of it being exactly 0 are very slim  To address the first point, you should smooth your histogram (e.g. using a uniform or Gaussian filter on the histogram itself). To solve the second point, instead of looking for  == 0, look for positive-to-negative (and viceversa) switching point.  To give you some minimal example of a possible approach: import numpy as np import matplotlib.pyplot as plt from scipy.ndimage import gaussian_filter1d np.random.seed(0) # generate noisy data raw = np.cumsum(np.random.normal(5, 100, 1000)) raw /= np.max(raw) # smooth smooth = gaussian_filter1d(raw, 100) # compute second derivative smooth_d2 = np.gradient(np.gradient(smooth)) # find switching points infls = np.where(np.diff(np.sign(smooth_d2)))[0] # plot results plt.plot(raw, label='Noisy Data') plt.plot(smooth, label='Smoothed Data') plt.plot(smooth_d2 / np.max(smooth_d2), label='Second Derivative (scaled)') for i, infl in enumerate(infls, 1): plt.axvline(x=infl, color='k', label=f'Inflection Point {i}') plt.legend(bbox_to_anchor=(1.55, 1.0))
Question: Suppose that I have the following observations of integers: df = pd.DataFrame({'observed_scores': [100, 100, 90, 85, 100, ...]})  I know that this can be used as an input to make a density plot: df['observed_scores'].plot.density()  but suppose that what I have is a counts table: df = pd.DataFrame({'observed_scores': [100, 95, 90, 85, ...], 'counts': [1534, 1399, 3421, 8764, ...})  which is cheaper to store than the full observed_scores Series (I have LOTS of observations). I know it's possible to plot the histogram using the counts, but how do I plot the density plot? If possible, can it be done without having to unstack/unravel the counts table into thousands of rows?

Answer: IIUC, statsmodels lets you fit a weighted KDE: from statsmodels.nonparametric.kde import KDEUnivariate df = pd.DataFrame({'observed_scores': [100, 95, 90, 85], 'counts': [1534, 1399, 3421, 8764]}) kde1= KDEUnivariate(df.observed_scores) kde_noweight = KDEUnivariate(df.observed_scores) kde1.fit(weights=df.counts, fft=False) kde_noweight.fit() plt.plot(kde1.support, kde1.density) plt.plot(kde_noweight.support, kde_noweight.density) plt.legend(['weighted', 'unweighted'])  Output:
Question: I have a pandas dataframe and want to select rows where values of a column starts with values of another column. I have tried the following: import pandas as pd df = pd.DataFrame({'A': ['apple', 'xyz', 'aa'], 'B': ['app', 'b', 'aa']}) df_subset = df[df['A'].str.startswith(df['B'])]  But it errors out and this solutions that I found also have not been helping. KeyError: "None of [Float64Index([nan, nan, nan], dtype='float64')] are in the [columns]"  np.where(df['A'].str.startswith(df['B']), True, False) from here also returns True for all.

Answer: For row wise comparison, we can use DataFrame.apply: m = df.apply(lambda x: x['A'].startswith(x['B']), axis=1) df[m] A B 0 apple app 2 aa aa  The reason your code is not working is because Series.str.startswith accepts a character sequence (a string scalar), and you are using a pandas Series. Quoting the docs:  pat : str Character sequence. Regular expressions are not accepted.
Question: I am trying to upload a package to PyPI using API tokens. I would like to use a project specific API token instead of an account specific token, as this seems more secure. However, since the project is not created on PyPI yet, there is no project for me to select when I try to create a new API token on the PyPI website. Since I have activated 2-factor authentication, I get an authentication error when trying to upload with twine. This is closely related to How to upload package to PyPi with Two Factor enabled?, except that the accepted answers does not address the particular issue of project versus account API tokens. I have also tried browsing through https://pypi.org/help/, but cannot seem to find any information there. So the question is then, how can I create an API token for a not-yet-created PyPI project?

Answer:  So the question is then, how can I create an API token for a not-yet-created PyPI project?  You cannot, for sure! Create and use a token for the account; later you can replace it with a project token.
Question: Today I had a python exam where following question was asked: Given the following code extract, Complete the code so the output is: 10 7 5. nums = list (range (?,?,?)) print(nums)  How is it possible to get such output in python using range function?

Answer: not sure if this answer the question, provided we can fill in any syntax to the ? as long it produce the result.  1st ? = 10 2nd ? = 4 3rd ? = -3))+(([5]   # nums = list(range( ? , ? , ? )) nums = list(range( 10 , 4 , -3))+(([5] )) print(nums) # nums = [10,7,5]
Question: Working in Python / pandas / dataframes I have these two dataframes: Dataframe one:  1 2 3 1 Stockholm 100 250 2 Stockholm 150 376 3 Stockholm 105 235 4 Stockholm 109 104 5 Burnley 145 234 6 Burnley 100 250  Dataframe two:  1 2 3 1 Stockholm 100 250 2 Stockholm 117 128 3 Stockholm 105 235 4 Stockholm 100 250 5 Burnley 145 234 6 Burnley 100 953  And I would like to find the duplicate rows found in Dataframe one and Dataframe two and remove the duplicates from Dataframe one. As in data frame two, you can find row 1, 3, 5 in data frame one, which would remove them from data frame on and create the below:  1 2 3 1 Stockholm 150 376 2 Stockholm 109 104 3 Burnley 100 250 

Answer: Use: df_merge = pd.merge(df1, df2, on=[1,2,3], how='inner') df1 = df1.append(df_merge) df1['Duplicated'] = df1.duplicated(keep=False) # keep=False marks the duplicated row with a True df_final = df1[~df1['Duplicated']] # selects only rows which are not duplicated. del df_final['Duplicated'] # delete the indicator column  The idea is as follows:  do a inner join on all the columns append the output of the inner join to df1 identify the duplicated rows in df1 select the not duplicated rows in df1  Each number corresponds to each line of code.
Question: I'm trying to login to Instagram using requests library. I succeeded using following script, however it doesn't work anymore. The password field becomes encrypted (checked the dev tools while logging in manually). I've tried : import re import requests from bs4 import BeautifulSoup link = 'https://www.instagram.com/accounts/login/' login_url = 'https://www.instagram.com/accounts/login/ajax/' payload = { 'username': 'someusername', 'password': 'somepassword', 'enc_password': '', 'queryParams': {}, 'optIntoOneTap': 'false' } with requests.Session() as s: r = s.get(link) csrf = re.findall(r"csrf_token\":\"(.*?)\"",r.text)[0] r = s.post(login_url,data=payload,headers={ "user-agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36", "x-requested-with": "XMLHttpRequest", "referer": "https://www.instagram.com/accounts/login/", "x-csrftoken":csrf }) print(r.status_code) print(r.url)  I found using dev tools: username: someusername enc_password: #PWD_INSTAGRAM_BROWSER:10:1592421027:ARpQAAm7pp/etjy2dMjVtPRdJFRPu8FAGILBRyupINxLckJ3QO0u0RLmU5NaONYK2G0jQt+78BBDBxR9nrUsufbZgR02YvR8BLcHS4uN8Gu88O2Z2mQU9AH3C0Z2NpDPpS22uqUYhxDKcYS5cA== queryParams: {"oneTapUsers":"[\"36990119985\"]"} optIntoOneTap: false  How can I login to Instagram using requests?

Answer: You can use authentication version 0 - plain password, no encryption: import re import requests from bs4 import BeautifulSoup from datetime import datetime link = 'https://www.instagram.com/accounts/login/' login_url = 'https://www.instagram.com/accounts/login/ajax/' time = int(datetime.now().timestamp()) payload = { 'username': '<USERNAME HERE>', 'enc_password': f'#PWD_INSTAGRAM_BROWSER:0:{time}:<PLAIN PASSWORD HERE>', # <-- note the '0' - that means we want to use plain passwords 'queryParams': {}, 'optIntoOneTap': 'false' } with requests.Session() as s: r = s.get(link) csrf = re.findall(r"csrf_token\":\"(.*?)\"",r.text)[0] r = s.post(login_url,data=payload,headers={ "user-agent": "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36", "x-requested-with": "XMLHttpRequest", "referer": "https://www.instagram.com/accounts/login/", "x-csrftoken":csrf }) print(r.status_code) print(r.url) print(r.text)  Prints: 200 https://www.instagram.com/accounts/login/ajax/ {"authenticated": true, "user": true, "userId": "XXXXXXXX", "oneTapPrompt": true, "reactivated": true, "status": "ok"}
Question: How can I have a member field of a dataclass same as the class name in python3.7+ ? I am trying to define a class like so (which can be done in Java or C++) -- which might be used as a class for LinkedList node @dataclass class Node: val:str next:Node prev:Node  However, all I get is NameError: name 'Node' is not defined. What should be the correct way to have self referential member variables in dataclasses

Answer: You need to add the following import to your file from __future__ import annotations  This enables deferred annotations, which are proposed in PEP 563. Deferred annotations allow you to reference a class that is not yet defined, in your example the Node class is not defined when you are still in it's body
Question: This is my view: def post_detail(request, year, month, day, slug): post = get_object_or_404(models.Post, slug=slug, status='published', publish__year=year, publish__month=month, publish__day=day) comment_form = forms.CommentForm() comments = post.comments.filter(active=True) context = { 'comments': comments, 'post': post, 'comment_form': comment_form, } return render(request, 'blog/post_detail.html', context)  Is there any way to calculate time of execution in Django ?

Answer: You can write a timer decorator to output the results in your console from functools import wraps import time def timer(func): """helper function to estimate view execution time""" @wraps(func) # used for copying func metadata def wrapper(*args, **kwargs): # record start time start = time.time() # func execution result = func(*args, **kwargs) duration = (time.time() - start) * 1000 # output execution time to console print('view {} takes {:.2f} ms'.format( func.__name__, duration )) return result return wrapper @timer def your_view(request): pass
Question: Until now I have been using conda as virtual environment and dependency management. However, some stuff does not work as expected when transfering my environment.yml file from my development machine to the production server. Now, I would like to look into alternatives. Poetry seems nice, especially because  poetry also maintains a lock file, and it has a benefit over pipenv because it keeps track of which packages are subdependencies. (https://realpython.com/effective-python-environment/#poetry)  which might improve stability quite a bit. However, I am working on science-heavy projects (matrices, data science, machine learning), so in practise I need the scipy stack (e.g. numpy, pandas, scitkit-learn).  Python became too slow for some pure computational workloads so numpy and scipy were born. [...] They are written in C and just wrapped as a python library. Compiling such libraries brings a set of challenges since they (more or less) have to be compiled on your machine for maximum performance and proper linking with libraries like glibc. Conda was introduced as an all-in-one solution to manage python environments for the scientific community. [...] Instead of using a fragile process of compiling libraries on your machine, libraries are precompiled and just downloaded when you request them. Unfortunately, the solution comes with a caveat - conda does not use PyPI, the most popular index of python packages. (https://modelpredict.com/python-dependency-management-tools#fnref:conda-compiling-challenges)  As far as I know, this doesn't even do Conda justice, because it does quite a bit of optimization to get the most out of my CPU/GPU/architecture for numpy. (https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/#Myth-#6:-Now-that-pip-uses-wheels,-conda-is-no-longer-necessary) https://numpy.org/install/ itself advises to use conda, but also says that one can install via pip (and poetry uses pypi)  For users who know, from personal preference or reading about the main differences between conda and pip below, they prefer a pip/PyPI-based solution, we recommend: [...] Use Poetry as the most well-maintained tool that provides a dependency resolver and environment management capabilities in a similar fashion as conda does.  I would like to get the stability of the poetry setup and the speed of the conda setup. How does poetry handle binary dependencies? Does it also, like conda, consider my hardware? If poetry not deliver in this regard, can I combine it with conda?

Answer: numpy provides several wheel files for different os, cpu architecture and python versions. wheel packages are precompiled, so the target system doesn't have to compile the package. poetry is able to choose the right wheel for you, depending on your system. Saying this, I would recommend using poetry, as long as you just need python packages, which are also available at pypi. As soon as you need other, non-python tools, stick to conda. (Disclaimer: I'm one of the maintainer of poetry). Also related: https://github.com/python-poetry/poetry/issues/1904
Question: Question Error highlighting and autoformatting can be great tools to help one create great notebooks. I am trying to change the settings on the VS code to allow me to autoformat to pep8 in my python notebooks. On this page for Jupiter notebooks have found that I have to put some lines in my .json files in the settings>preference of VSCode in order to do this. I am particularly interested in changing my code to the pep8 coding convention and also adding linting in order to highlight errors.  linting (error highlighting) autoformatting (autopep8)  I am using VS Code on Ubuntu 18.04. Below is my attempt that led to an error "Code language not supported or defined". Attempt After installing the Python extension and the autopep8 extension in VS code and running pip3 install autopep8  I got an error message and was unable to use pep8.  If you may know how to set up an efficient working environment in VS Code for Jupyter notebooks I would really appreciate any assistance Summary How to set up:  linting (error highlighting) autoformatting (autopep8)  in VS code for python notebooks. Edit 1: I also tried running autopep8 in the command palette and got the error Command 'autopep8' resulted in an error (Running the contributed command: 'extension.sayHello' failed.) 

Answer: Nbextensions are notebook extensions and only work within the notebook itself. VS Code does not support native notebooks so these extensions won't work at the time. They are planning to add it in future releases per link
Question: I have a django project and i want to create the .pyc files and remove the source code. My project folder name is mysite and I ran the command python -m compileall mysite. The .pyc files are created. After that i tried to run my project with python __pycache__/manage.cpython-37.pyc runserver command but i've got an error such as ModuleNotFoundError: No module named 'mysite' There are two things I would like to ask about this. First, how can I solve this problem and run my project with .pyc file? Secondly, is it enough to move the .pyc files created to a separate folder in accordance with the django project structure?

Answer: First of all, I created a new folder in another directory such as a new django project and i created my app folders, static folder, templates folder etc. manually as the same as my django project architecture that I created before. Then, I moved the .pyc files that I created with the compileall command to my new project folders. As you know, while creating .pyc files, a .cpython-37 section is added to the file names automatically (for example, manage.py -> manage.cpython-37.pyc). I removed that section and i converted them to manage.pyc, views.pyc, etc. So my file structure was like this: mysite/ manage.pyc mysite/ __init__.pyc settings.pyc urls.pyc wsgi.pyc app/ migrations/ __init__.pyc __init__.pyc admin.pyc apps.pyc models.pyc tests.pyc views.pyc ...  After I created this django project structure with .pyc files, i ran the python manage.pyc runserver command and it works.
Question: I have a database table called Person contains following columns:  Id, first_name, last_name,  So is there any way to assign different name to table fields in django model. like this class Person(models.Model): firstname = models.CharField(max_length = 30) lastname = models.CharField(max_length = 30)  firstname instead of first_name and lastname instead of last_name

Answer: You can pass db_column to the field to customise the column name for a field class Person(models.Model): firstname = models.CharField(max_length=30, db_column='first_name') lastname = models.CharField(max_length=30, db_column='last_name')
Question: There are 2 files named compare 1.txt and compare2.txt having random numbers in non-sequential order cat compare1.txt 57 11 13 3 889 014 91  cat compare2.txt 003 889 13 14 57 12 90  Aim  Output list of all the numbers which are present in compare1 but not in compare 2 and vice versa  If any number has zero in its prefix, ignore zeros while comparing ( basically the absolute value of number must be different to be treated as a mismatch ) Example - 3 should be considered matching with 003 and 014 should be considered matching with 14, 008 with 8 etc   Note - It is not necessary that matching must necessarily happen on the same line. A number present in the first line in compare1 should be considered matched even if that same number is present on other than the first line in compare2 Expected output 90 91 12 11  PS ( I don't necessarily need this exact order in expected output, just these 4 numbers in any order would do ) What I tried? Obviously I didn't have hopes of getting the second condition correct, I tried only fulfilling the first condition but couldn't get correct results. I had tried these commands grep -Fxv -f compare1.txt compare2.txt && grep -Fxv -f compare2.txt compare1.txt  cat compare1.txt compare2.txt | sort |uniq  Edit - A Python solution is also fine

Answer: Could you please try following, written and tested with shown samples in GNU awk. awk ' { $0=$0+0 } FNR==NR{ a[$0] next } ($0 in a){ b[$0] next } { print } END{ for(j in a){ if(!(j in b)){ print j } } } ' compare1.txt compare2.txt  Explanation: Adding detailed explanation for above. awk ' ##Starting awk program from here. { $0=$0+0 ##Adding 0 will remove extra zeros from current line,considering that your file doesn't have float values. } FNR==NR{ ##Checking condition FNR==NR which will be TRUE when 1st Input_file is being read. a[$0] ##Creating array a with index of current line here. next ##next will skip all further statements from here. } ($0 in a){ ##Checking condition if current line is present in a then do following. b[$0] ##Creating array b with index of current line. next ##next will skip all further statements from here. } { print } ##will print current line from 2nd Input_file here. END{ ##Starting END block of this code from here. for(j in a){ ##Traversing through array a here. if(!(j in b)){ print j } ##Checking condition if current index value is NOT present in b then print that index. } } ' compare1.txt compare2.txt ##Mentioning Input_file names here.
Question: My question is very simple , as you read the title I want plugin similar to GitLens that I found in vscode. As you know with GitLens you can easily see the difference between two or multiple commits. I searched it up and I found GitToolBox but I don't know how to install it as well and I don't think that's like GitLens...

Answer: You can use Git Toolbox link here. Features :  Git status:  number of ahead / behind commits for current branch as status bar widget ahead / behind, current branch, tags on HEAD as Project View decoration on modules status bar widget with detailed information and additional actions  Git blame:  inline blame - show blame for line at caret in active editor
Question: I wanted to join two dictionaries using | operator and I got the following error: TypeError: unsupported operand type(s) for |: 'dict' and 'dict'  The MWE code is the following: d1 = {'k': 1, 'l': 2, 'm':4} d2 = {'g': 3, 'm': 7} e = d1 | d2 

Answer: The merge (|) and update (|=) operators for dictionaries were introduced in Python 3.9 so they do not work in older versions. You have an option to either update your Python interpreter to Python 3.9 or use one of the alternatives: # option 1: e = d1.copy() e.update(d2) # option 2: e = {**d1, **d2}  However, should you want to update to Python 3.9 you can save some memory updating dictionary d1 directly instead of creating another dictionary using in-place merge operation: d1 |= d2  Which is equivalent of the following in the older versions of Python: d1.update(d2)
Question: How can I generate a new column listing repeated values? For example, my dataframe is: id color 123 white 123 white 123 white 345 blue 345 blue 678 red  This is the desired output: # id color 1 123 white 1 123 white 1 123 white 2 345 blue 2 345 blue 3 678 red 

Answer: Check withfactorize df['#']=df.id.factorize()[0]+1 df id color # 0 123 white 1 1 123 white 1 2 123 white 1 3 345 blue 2 4 345 blue 2 5 678 red 3  Another method df.groupby('id').ngroup()+1 0 1 1 1 2 1 3 2 4 2 5 3 dtype: int64  To add it to the first positon: df.insert(loc=0, column='#', value=df.id.factorize()[0]+1) df # id color 0 1 123 white 1 1 123 white 2 1 123 white 3 2 345 blue 4 2 345 blue 5 3 678 red
Question: I want to find if there are two of the same digits next to each other, and the digit behind and in front of the pair is different. For example, 123456678 should match as there is a double 6, 1234566678 should not match as there is no double with different surrounding numbers. 12334566 should match because there are two 3s. So far i have this which works only with 1, and as long as the double is not at the start or end of the string, however I can deal with that by adding a letter at the start and end. ^.*([^1]11[^1]).*$  I know I can use [0-9] instead of the 1s but the problem is having them all be the same digit.

Answer: With regex, it is much more convenient to use a PyPi regex module with the (*SKIP)(*FAIL) based pattern: import regex rx = r'(\d)\1{2,}(*SKIP)(*F)|(\d)\2' l = ["123456678", "1234566678"] for s in l: print(s, bool(regex.search(rx, s)) )  See the Python demo. Output: 123456678 True 1234566678 False  Regex details  (\d)\1{2,}(*SKIP)(*F) - a digit and then two or more occurrences of the same digit | - or (\d)\2 - a digit and then the same digit.  The point is to match all chunks of identical 3 or more digits and skip them, and then match a chunk of two identical digits. See the regex demo.