eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Recently my jupyter notebook stopped launching. When I try the command jupyter notebook from anaconda prompt but it gives error  Traceback (most recent call last): File "C:\Users\Dell\anaconda3\Scripts\jupyter-notebook-script.py", line 6, in from notebook.notebookapp import main File "C:\Users\Dell\anaconda3\lib\site-packages\notebook\notebookapp.py", line 51, in from zmq.eventloop import ioloop File "C:\Users\Dell\anaconda3\lib\site-packages\zmq_init_.py", line 50, in from zmq import backend File "C:\Users\Dell\anaconda3\lib\site-packages\zmq\backend_init_.py", line 40, in reraise(*exc_info) File "C:\Users\Dell\anaconda3\lib\site-packages\zmq\utils\sixcerpt.py", line 34, in reraise raise value File "C:\Users\Dell\anaconda3\lib\site-packages\zmq\backend_init_.py", line 27, in ns = select_backend(first) File "C:\Users\Dell\anaconda3\lib\site-packages\zmq\backend\select.py", line 28, in select_backend mod = import(name, fromlist=public_api) File "C:\Users\Dell\anaconda3\lib\site-packages\zmq\backend\cython_init.py", line 6, in from . import (constants, error, message, context, ImportError: DLL load failed while importing error: The specified module could not be found.  I even tried reinstalling anaconda and upgraded to python 3.8.3 on windows 10 but still get the same error. When I tried to check jupyter notebook's version it said that ipykernel and some other things were not installed. jupyter --version gives me this:  jupyter core : 4.6.3 jupyter-notebook : 6.1.1 qtconsole : 4.7.6 ipython : 7.18.1 ipykernel : not installed jupyter client : not installed jupyter lab : not installed nbconvert : 5.6.1  But installing ipykernel with conda install ipykernel says  All requested packages already installed.  I also tried ipython kernel install --name <env_name> --user but this gives another dll error. Reading some other problems in stackoverflow I went and checked my enviroment variables.   Is there a problem with this environment variable. Please help. Screenshot of the anaconda prompt with error.  Edit: The anaconda navigator does not launch either. anaconda-navigator on the anaconda prompt gives another error:  I also tried: conda install qt --force conda install pyqt --force  But that did'nt help. Does previously installed anaconda cause such error?

Answer: I found what I did wrong (silly me). Microsoft visual C++ 2015-2019 was somehow removed when I tried to install openCV manually. Didn't think that such an install would make such big impact, have to keep that in mind now but installing the latest solved all the problems. P.S.: This solution might not work for someone else with similar problem. But its worth taking a note.
Question: In Pandas, How can I check how sparse a DataFrame? Is there any function available, or I will need to write my own? For now, I have this: df = pd.DataFrame({'a':[1,0,1,1,3], 'b':[0,0,0,0,1], 'c':[4,0,0,0,0], 'd':[0,0,3,0,0]})   a b c d 0 1 0 4 0 1 0 0 0 0 2 1 0 0 3 3 1 0 0 0 4 3 1 0 0  sparsity = sum((df == 0).astype(int).sum())/df.size  Which divides the number of zeros by the total number of elements, in this example it's 0.65. Wanted to know if there is any better way to do this. And if there is any function which gives more information about the sparsity (like NaNs, any other prominent number like -1).

Answer: One idea for your solution is convert to numpy array, compare and use mean: a = (df.to_numpy() == 0).mean() print (a) 0.65  If want use Sparse dtypes is possible use: #convert each column to SparseArray sparr = df.apply(pd.arrays.SparseArray) print (sparr) a b c d 0 1 0 4 0 1 0 0 0 0 2 1 0 0 3 3 1 0 0 0 4 3 1 0 0 print (sparr.dtypes) a Sparse[int64, 0] b Sparse[int64, 0] c Sparse[int64, 0] d Sparse[int64, 0] dtype: object print (sparr.sparse.density) 0.35
Question: I have this problem: I hav this code that is trying to count bigrams in a text file. An if statement checks wether the tuple is in a dictionary. If it is, the value (counter) is one-upped. If it doesn't exist, the code shoud create a key-value pair with the tuple as key and the value 1. for i in range(len(temp_list)-1): temp_tuple=(temp_list[i], temp_list[i+1]) if bigramdict[temp_tuple] in bigramdict: bigramdict[temp_tuple] = bigramdict[temp_tuple]+1 else: bigramdict[temp_tuple] = 1  However, whenever I run the code, it throws a KeyError on the very first tuple. As far as I understand, KeyError gets thrown when a key in dict doesn't exist, which is the case here. That's why I have the if statement to see if there is a key. Normally, the program should see that there is no key and go to the else to create one. However, it gets stuck on the if and complains about the missing key. Why does it not recognize that this is a conditional statement? Pls help.

Answer: What you were trying to do was if temp_tuple in bigramdict:  instead of if bigramdict[temp_tuple] in bigramdict:
Question: I would like to know how to stop lightgbm logging. What kind of settings should I use to stop the log? Also, is there a way to output only your own log with the lightgbm log stopped?

Answer: I think you can disable lightgbm logging using verbose=-1 in both Dataset constructor and train function, as mentioned here
Question: I know that the title is very confusing, so let me take the Binary Search Tree as an example: Using ordinary class definition # This code passed mypy test from typing import Generic, TypeVar T = TypeVar('T') class BST(Generic[T]): class Node: def __init__( self, val: T, left: 'BST.Node', right: 'BST.Node' ) -> None: self.val = val self.left = left self.right = right  The above code passed mypy test. Using dataclass However, when I tried to use dataclass to simplify the definition of Node, the code failed in mypy test. # This code failed to pass mypy test from dataclasses import dataclass from typing import Generic, TypeVar T = TypeVar('T') class BST(Generic[T]): @dataclass class Node: val: T left: 'BST.Node' right: 'BST.Node'  mypy gave me this error message: (test_typing.py:8 is the line val: T) test_typing.py:8: error: Type variable "test_typing.T" is unbound test_typing.py:8: note: (Hint: Use "Generic[T]" or "Protocol[T]" base class to bind "T" inside a class) test_typing.py:8: note: (Hint: Use "T" in function signature to bind "T" inside a function)  Pinpoint the problem # This code passed mypy test, suggest the problem is the reference to `T` in the dataclass definition from dataclasses import dataclass from typing import Generic, TypeVar T = TypeVar('T') class BST(Generic[T]): @dataclass class Node: val: int # chose `int` just for testing left: 'BST.Node' right: 'BST.Node'  The above code agained passed the test, so I think the problem is the reference to T in the dataclass definition. Does anyone know how to future fix this to meet my original goal?

Answer: Let's start with what is written in PEP 484 about scoping rules for type variables:  A generic class nested in another generic class cannot use same type variables. The scope of the type variables of the outer class doesn't cover the inner one: T = TypeVar('T') S = TypeVar('S') class Outer(Generic[T]): class Bad(Iterable[T]): # Error ... class AlsoBad: x = None # type: List[T] # Also an error class Inner(Iterable[S]): # OK ... attr = None # type: Inner[T] # Also OK   This is why your example with nested decorated class does not work. Now let's answer the question why the example works with __init__ function that takes TypeVar variable. This is because the method __init__ is treated by mypy as a generic method with an independent TypeVar variable. For example reveal_type(BST[int].Node.__init__) shows Revealed type is 'def [T, T] (self: main.BST.Node, val: T'-1, left: main.BST.Node, right: main.BST.Node)'. i.e. T is not bound to int here.
Question: I want a code that deletes all instances of any number that has been repeated from a list. E.g.: Inputlist = [2, 3, 6, 6, 8, 9, 12, 12, 14] Outputlist = [2,3,8,9,14]  I have tried to remove the duplicated elements in the list already (by using the "unique" function), but it leaves a single instance of the element in the list nevertheless! seen = set() uniq = [] for x in Outputlist: if x not in seen: uniq.append(x) seen.add(x) seen  I went through a lot of StackOverflow articles too, but all of them differ in the idea that they are searching for removing common elements from two different lists, or that they want just one instance of each element to still be kept. I want to simply remove all common elements.

Answer: You can use a Counter >>> from collections import Counter >>> l = [2, 3, 6, 6, 8, 9, 12, 12, 14] >>> res = [el for el, cnt in Counter(l).items() if cnt==1] >>> res [2, 3, 8, 9, 14]
Question: i am trying to take diff value from previous row in a dataframe by grouping column "group", there are several similar questions but i can't get this working.  date group value 0 2020-01-01 A 808 1 2020-01-01 B 331 2 2020-01-02 A 612 3 2020-01-02 B 1391 4 2020-01-03 A 234 5 2020-01-04 A 828 6 2020-01-04 B 820 6 2020-01-05 A 1075 8 2020-01-07 B 572 9 2020-01-10 B 736 10 2020-01-10 A 1436 df.sort_values(['group','date'], inplace=True) df['diff'] = df['value'].diff() print(df) date value group diff 1 2020-01-03 234 A NaN 8 2020-01-01 331 B 97.0 2 2020-01-07 572 B 241.0 9 2020-01-02 612 A 40.0 5 2020-01-10 736 B 124.0 17 2020-01-01 808 A 72.0 14 2020-01-04 820 B 12.0 4 2020-01-04 828 A 8.0 18 2020-01-05 1075 A 247.0 7 2020-01-02 1391 B 316.0 10 2020-01-10 1436 A 45.0  This is the result that i need  date group value diff 0 2020-01-01 A 808 Na 2 2020-01-02 A 612 -196 4 2020-01-03 A 234 -378 5 2020-01-04 A 828 594 6 2020-01-05 A 1075 247 10 2020-01-10 A 1436 361 1 2020-01-01 B 331 Na 3 2020-01-02 B 1391 1060 6 2020-01-04 B 820 -571 8 2020-01-07 B 572 -248 9 2020-01-10 B 736 164 

Answer: Shifts through each group to create a calculated column. Subtract that column from the original value column to create the difference column. df.sort_values(['group','date'], ascending=[True,True], inplace=True) df['shift'] = df.groupby('group')['value'].shift() df['diff'] = df['value'] - df['shift'] df = df[['date','group','value','diff']] 1 df date group value diff 0 2020-01-01 A 808 NaN 2 2020-01-02 A 612 -196.0 4 2020-01-03 A 234 -378.0 5 2020-01-04 A 828 594.0 6 2020-01-05 A 1075 247.0 10 2020-01-10 A 1436 361.0 1 2020-01-01 B 331 NaN 3 2020-01-02 B 1391 1060.0 6 2020-01-04 B 820 -571.0 8 2020-01-07 B 572 -248.0 9 2020-01-10 B 736 164.0
Question: I am trying to access my webcam with cv2(opencv-python) package. When I try to import it I get this error: Traceback (most recent call last): File "server.py", line 6, in <module> import cv2 File "/usr/local/lib/python3.8/dist-packages/cv2/__init__.py", line 5, in <module> from .cv2 import * ImportError: libGL.so.1: cannot open shared object file: No such file or directory  Note: I am trying to import this package on putty, on Linode server - that might be useful information. If anyone can explain to me what is happening and maybe solve the problem I will highly appreciate it!

Answer: Install opencv-python-headless instead of opencv-python. Server (headless) environments do not have GUI packages installed which is why you are seeing the error. opencv-python depends on Qt which in turn depends on X11 related libraries. Other alternative is to run sudo apt-get install -y libgl1-mesa-dev which will provide the missing libGL.so.1 if you want to use opencv-python. The libgl1-mesa-dev package might be named differently depending on your GNU/Linux distribution. Full installation guide for opencv-python can be found from the package documentation: https://github.com/skvark/opencv-python#installation-and-usage
Question: I'm trying python coroutine programming using asyncio. This is my code. import asyncio async def coro_function(): return 2 + 2 async def get(): return await coro_function() print(asyncio.iscoroutinefunction(get)) loop = asyncio.get_event_loop() a1 = loop.create_task(get) loop.run_until_complete(a1)  But when I execute it, it gives me error True Traceback (most recent call last): File "a.py", line 13, in <module> a1 = loop.create_task(get) File "/home/alie/anaconda3/lib/python3.7/asyncio/base_events.py", line 405, in create_task task = tasks.Task(coro, loop=self) TypeError: a coroutine was expected, got <function get at 0x7fe1280b6c80>  How to solve it?

Answer: You're passing in the function get. In order to pass in a coroutine, pass in get(). a1 = loop.create_task(get()) loop.run_until_complete(a1)  Take a look at the types: >>> type(get) <class 'function'> >>> print(type(get())) <class 'coroutine'>  get is a coroutine function, i.e. a function that returns a coroutine object, get(). For more information and a better understanding of the fundamentals, take a look at the docs.
Question: The FashionMNIST dataset has 10 different output classes. How can I get a subset of this dataset with only specific classes? In my case, I only want images of sneaker, pullover, sandal and shirt classes (their classes are 7,2,5 and 6 respectively). This is how I load my dataset. train_dataset_full = torchvision.datasets.FashionMNIST(data_folder, train = True, download = True, transform = transforms.ToTensor()) The approach I’ve followed is below. Iterate through the dataset, one by one, then compare the 1st element (i.e. class) in the returned tuple to my required class. I’m stuck here. If the value returned is true, how can I append/add this observation to an empty dataset? sneaker = 0 pullover = 0 sandal = 0 shirt = 0 for i in range(60000): if train_dataset_full[i][1] == 7: sneaker += 1 elif train_dataset_full[i][1] == 2: pullover += 1 elif train_dataset_full[i][1] == 5: sandal += 1 elif train_dataset_full[i][1] == 6: shirt += 1  Now, in place of sneaker += 1, pullover += 1, sandal += 1 and shirt += 1 I want to do something like this empty_dataset.append(train_dataset_full[i]) or something similar. If the above approach is incorrect, please suggest another method.

Answer: Finally found the answer. dataset_full = torchvision.datasets.FashionMNIST(data_folder, train = True, download = True, transform = transforms.ToTensor()) # Selecting classes 7, 2, 5 and 6 idx = (dataset_full.targets==7) | (dataset_full.targets==2) | (dataset_full.targets==5) | (dataset_full.targets==6) dataset_full.targets = dataset_full.targets[idx] dataset_full.data = dataset_full.data[idx]
Question: Looking to sharpen my data science skills. I am practicing url data pulls from a sports site and the json file has multiple nested dictionaries. I would like to be able to pull this data to map my own custom form of the leaderboard in matplotlib, etc., but am having a hard time getting the json to a workable df. The main website is: https://www.usopen.com/scoring.html Looking at the background I believe the live info is being pulled from the link listed in the short code below. I'm working in Jupyter notebooks. I can get the data successfully pulled. But as you can see, it is pulling multiple nested dictionaries which is making it very difficult in getting a simple dataframe pulled. Was just looking to get player, score to par, total, and round pulled. Any help would be greatly appreciated, thank you! import pandas as pd import urllib as ul import json url = "https://gripapi-static-pd.usopen.com/gripapi/leaderboard.json" response = ul.request.urlopen(url) data = json.loads(response.read()) print(data) 

Answer: Simple and Quick Solution. A better solution might exist with JSON normalize from pandas but this is fairly good for your use case. def func(x): if not any(x.isnull()): return (x['round'], x['player']['firstName'], x['player']['identifier'], x['toParToday']['value'], x['totalScore']['value']) df = pd.DataFrame(data['standings']) df['round'] = data['currentRound']['name'] df = df[['player', 'toPar', 'toParToday', 'totalScore', 'round']] info = df.apply(func, axis=1) info_df = pd.DataFrame(list(info.values), columns=['Round', 'player_name', 'pid', 'to_par_today', 'totalScore']) info_df.head()
Question: I have 50 variables in my dataframe. 46 are dependent variables and 4 are independent variables (precipitation, temperature, dew, snow). I want to calculate the mutual information of my dependent variables against my independent. So in the end I want a dataframe like this  Right now I am calculating it using the following but it's taking so long because I have to change my y each time: X = df[['Temperature', 'Precipitation','Dew','Snow']] # Features y = df[['N0037']] #target from sklearn.feature_selection import mutual_info_regression mi = mutual_info_regression(X, y) mi /= np.max(mi) mi = pd.Series(mi) mi.index = X.columns mi.sort_values(ascending=False) print(mi)  How to calculate pairwise mutual information for the entire dataset?

Answer: Using list comprehension: indep_vars = ['Temperature', 'Precipitation', 'Dew', 'Snow'] # set independent vars dep_vars = df.columns.difference(indep_vars).tolist() # set dependent vars from sklearn.feature_selection import mutual_info_regression as mi_reg df_mi = pd.DataFrame([mi_reg(df[indep_vars], df[dep_var]) for dep_var in dep_vars], index = dep_vars, columns = indep_vars).apply(lambda x: x / x.max(), axis = 1)
Question: I need to check if a point lies inside a bounding cuboid. The number of cuboids is very large (~4M). The code I come up with is: import numpy as np # set the numbers of points and cuboids n_points = 64 n_cuboid = 4000000 # generate the test data points = np.random.rand(1, 3, n_points)*512 cuboid_min = np.random.rand(n_cuboid, 3, 1)*512 cuboid_max = cuboid_min + np.random.rand(n_cuboid, 3, 1)*8 # main body: check if the points are inside the cuboids inside_cuboid = np.all((points > cuboid_min) & (points < cuboid_max), axis=1) indices = np.nonzero(inside_cuboid)  It takes 8 seconds to run np.all and 3 seconds to run np.nonzero on my computer. Any idea to speed up the code?

Answer: We can reduce memory congestion for all-reduction with slicing along the smallest axis length of 3 to get inside_cuboid - out = (points[0,0,:] > cuboid_min[:,0]) & (points[0,0,:] < cuboid_max[:,0]) & \ (points[0,1,:] > cuboid_min[:,1]) & (points[0,1,:] < cuboid_max[:,1]) & \ (points[0,2,:] > cuboid_min[:,2]) & (points[0,2,:] < cuboid_max[:,2])  Timings - In [43]: %timeit np.all((points > cuboid_min) & (points < cuboid_max), axis=1) 2.49 s ± 20 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) In [51]: %%timeit ...: out = (points[0,0,:] > cuboid_min[:,0]) & (points[0,0,:] < cuboid_max[:,0]) & \ ...: (points[0,1,:] > cuboid_min[:,1]) & (points[0,1,:] < cuboid_max[:,1]) & \ ...: (points[0,2,:] > cuboid_min[:,2]) & (points[0,2,:] < cuboid_max[:,2]) 1.95 s ± 10.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Question: I have been struggling to find the pandas solution to this without looping: input: df = pd.DataFrame({'A' : [[6,1,1,1], [1,5,1,1], [1,1,11,1], [1,1,1,20]]}) A 0 [6, 1, 1, 1] 1 [1, 5, 1, 1] 2 [1, 1, 11, 1] 3 [1, 1, 1, 20]  output:  A B 0 [6, 1, 1, 1] 6 1 [1, 5, 1, 1] 5 2 [1, 1, 11, 1] 11 3 [1, 1, 1, 20] 20  I have tried so many different things over the past hour or so, and I know the solution will be an embarrassingly simple one-liner. Thanks for your help -- not my python day today!

Answer: You can do a simple list comprehension: df['B'] = [s[i] for i, s in zip(df.index, df['A'])]  Or if you want only diagonal values: df['B'] = np.diagonal([*df['A']])    A B 0 [6, 1, 1, 1] 6 1 [1, 5, 1, 1] 5 2 [1, 1, 11, 1] 11 3 [1, 1, 1, 20] 20
Question:  Hi Guys, I am trying to map the district shapefile into assembly constituencies. I have shape files for [Both].Basically I have to map all the variables given at district level in census data to assembly constituency level. So I am following a pycon talk. Everything is working fine but I am getting error in get_intersection function.Error for that is TopologicalError: The operation 'GEOSIntersection_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.polygon.Polygon object at 0x7f460250ce10>.  I have tried to use both pygeos and rtree. There were some links which said that problem is in pygeos. So,I used rtree.But of no avail.Please help Thanks in advance. Code tried by me is constituencies=gpd.GeoDataFrame.from_file('/content/AC_All_Final.shp') districts=gpd.GeoDataFrame.from_file('/content/2001_Dist.shp') districts['AREA'] = districts.geometry.area constituencies['AREA'] = constituencies.geometry.area merged = gpd.sjoin(districts, constituencies).reset_index().rename( columns={'index': 'index_left'}) def get_intersection(row): left_geom = districts['geometry'][row['index_left']] right_geom = constituencies['geometry'][row['index_right']] return left_geom.intersection(right_geom) ***Error is at this point*** merged['geometry'] = merged.apply(get_intersection, axis=1) merged['AREA'] = merged.geometry.area  Error trace is given below: TopologyException: Input geom 1 is invalid: Ring Self-intersection at or near point 77.852561819157373 14.546596140487276 at 77.852561819157373 14.546596140487276 --------------------------------------------------------------------------- TopologicalError Traceback (most recent call last) <ipython-input-17-8123669e025c> in <module>() 4 return left_geom.intersection(right_geom) 5 ----> 6 merged['geometry'] = merged.apply(get_intersection, axis=1) 7 merged['AREA'] = merged.geometry.area 7 frames /usr/local/lib/python3.6/dist-packages/shapely/topology.py in _check_topology(self, err, *geoms) 36 "The operation '%s' could not be performed. " 37 "Likely cause is invalidity of the geometry %s" % ( ---> 38 self.fn.__name__, repr(geom))) 39 raise err 40 TopologicalError: The operation 'GEOSIntersection_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.polygon.Polygon object at 0x7f460250ce10> 

Answer: The error message tells you exactly what is going on. Some of your geometries are not valid, so you have to make them valid before doing your apply. The simple trick, which works in most of the cases is using buffer(0). merged['geometry'] = merged.buffer(0)  Since the issue is with geometry validity and is raised by GEOS, it does not matter if you use shapely/rtree backend or pygeos.
Question: I have a [10,10] numpy.ndarray. I am trying to get the index the second highest number in each row. So for the array: [101 0 1 0 0 0 1 1 2 0] [ 0 116 1 0 0 0 0 0 1 0] [ 1 4 84 2 2 0 2 4 6 1] [ 0 2 0 84 0 6 0 2 3 0] [ 0 0 1 0 78 0 0 2 0 11] [ 2 0 0 1 1 77 5 0 2 0] [ 1 2 1 0 1 2 94 0 1 0] [ 0 1 1 0 0 0 0 96 0 4] [ 1 5 4 3 1 3 0 1 72 4] [ 0 1 0 0 3 2 0 7 0 82]  Expected result: [8, 2, 8, 5, 9, ...] Any suggestions?

Answer: The amazing numpy.argsort() function makes this task really simple. Once the sorted indices are found, get the second to last column. m = np.array([[101, 0, 1, 0, 0, 0, 1, 1, 2, 0], [ 0, 116, 1, 0, 0, 0, 0, 0, 1, 0], [ 1, 4, 84, 2, 2, 0, 2, 4, 6, 1], [ 0, 2, 0, 84, 0, 6, 0, 2, 3, 0], [ 0, 0, 1, 0, 78, 0, 0, 2, 0, 11], [ 2, 0, 0, 1, 1, 77, 5, 0, 2, 0], [ 1, 2, 1, 0, 1, 2, 94, 0, 1, 0], [ 0, 1, 1, 0, 0, 0, 0, 96, 0, 4], [ 1, 5, 4, 3, 1, 3, 0, 1, 72, 4], [ 0, 1, 0, 0, 3, 2, 0, 7, 0, 82]]) # Get index for the second highest value. m.argsort()[:,-2]  Output: array([8, 8, 8, 5, 9, 6, 5, 9, 1, 7], dtype=int32)