eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a small dataframe, consisting of just two columns, which should have all floats in it. So, I have two fields name 'Price' and 'Score'. When I look at the data, it all looks like floats to me, but apparently something is a string. Is there some way to kick out these things that are strings, but look like floats? Or, is there a way to force everything to be float? The error occurs on the last line show here, and then nothing else works. df = pd.read_csv('C:\\my_path\\analytics.csv') print('done!') modDF = df[['Price', 'Score']].copy() modDF = modDF[:100] for i_dataset, dataset in enumerate(datasets): X, y = dataset # normalize dataset for easier parameter selection X = StandardScaler().fit_transform(X)  Here is the Stack Trace: datasets = [modDF] for i_dataset, dataset in enumerate(datasets): X, y = dataset # normalize dataset for easier parameter selection X = StandardScaler().fit_transform(X) Traceback (most recent call last): File "<ipython-input-18-013c2a6bef49>", line 5, in <module> X = StandardScaler().fit_transform(X) File "C:\Users\rs\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\base.py", line 553, in fit_transform return self.fit(X, **fit_params).transform(X) File "C:\Users\rs\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\preprocessing\data.py", line 639, in fit return self.partial_fit(X, y) File "C:\Users\rs\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\preprocessing\data.py", line 663, in partial_fit force_all_finite='allow-nan') File "C:\Users\rs\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\utils\validation.py", line 496, in check_array array = np.asarray(array, dtype=dtype, order=order) File "C:\Users\rs\AppData\Local\Continuum\anaconda3\lib\site-packages\numpy\core\numeric.py", line 538, in asarray return array(a, dtype, copy=False, order=order) ValueError: could not convert string to float: 'Price' 

Answer: You could try using pd.to_numeric like so: df = df.apply(pd.to_numeric, errors='coerce', downcast='float')  Which would try to convert your data to float, and the data that is not float will be returned as Nan. Then df.dropna(how='any', axis=0, inplace=True)  which just drops any rows with at least 1 Nan value in it.
Question: I have updated to djangorestframework==3.11.0 from older version. Now I've got this error,  TypeError: register() got an unexpected keyword argument 'base_name'  Traceback ... ... ... File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/abu/projects/django-example/django2x/urls.py", line 21, in <module> path('sample/', include('sample.urls')), File "/home/abu/.virtualenvs/django-example/lib/python3.6/site-packages/django/urls/conf.py", line 34, in include urlconf_module = import_module(urlconf_module) File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 994, in _gcd_import File "<frozen importlib._bootstrap>", line 971, in _find_and_load File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 665, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/abu/projects/django-example/sample/urls.py", line 6, in <module> router.register(r'musician', MusicianViewset, base_name='musician') TypeError: register() got an unexpected keyword argument 'base_name' 

Answer: From the release notes of Django RestFramework and DRF 3.9 announcement they mentioned that  Deprecate the Router.register base_name argument in favor of basename. #5990   Which means, the argument base_name is no longer available from DRF=3.11 onwards and use basename instead So, Change your router config as, router.register(r'musician', MusicianViewset, basename='musician') router.register(r'album', AlbumViewset, basename='album')
Question: Hi I am new to python and keras and looking in some keran example code: model = VGG19(weights='imagenet',include_top=False) model.trainable=False x = Flatten(name='flat')(model) x = Dense(512, activation='relu', name='fc1')(x) x = Dense(512, activation='relu', name='fc2')(x) x = Dense(10, activation='softmax', name='predictions')(x)  and just don't understand x=Dense(....)(x) what is second paranteses usage? and what is this syntax name if i want to see python doc's?

Answer: This might be an easier to understand version of these chained calls: model = VGG19(weights='imagenet',include_top=False) model.trainable=False layer1 = Flatten(name='flat')(model) layer2 = Dense(512, activation='relu', name='fc1')(layer1) layer3 = Dense(512, activation='relu', name='fc2')(layer2) layer4 = Dense(10, activation='softmax', name='predictions')(layer3)  which could be rewritten as: model = VGG19(weights='imagenet',include_top=False) model.trainable=False model.add( Flatten(name='flat')) model.add( Dense(512, activation='relu', name='fc1')) model.add( Dense(512, activation='relu', name='fc2')) model.add( Dense(10, activation='softmax', name='predictions'))
Question: Imagine I have a set of functions like below. foo has a lot of arguments of various types, and bar passes all its arguments to that other function. Is there any way to make mypy understand that bar has the same type as foo without explicitly copying the whole argument list? def foo(a: int, b: float, c: str, d: bool, *e: str, f: str = "a", g: str = "b") -> str: ... def bar(*args, **kwargs): val = foo(*args, **kwargs) ... return val 

Answer: There's been a lot of discussion about adding this feature here. For the straightforward case of passing all arguments you can use the recipe from this comment: F = TypeVar('F', bound=Callable[..., Any]) class copy_signature(Generic[F]): def __init__(self, target: F) -> None: ... def __call__(self, wrapped: Callable[..., Any]) -> F: ... def f(x: bool, *extra: int) -> str: ... @copy_signature(f) def test(*args, **kwargs): return f(*args, **kwargs) reveal_type(test) # Revealed type is 'def (x: bool, *extra: int) -> str'
Question: What is the parameter alpha in ridge regression and how does it influence the trained regression? So examples would be helpful for me :)

Answer: Ridge or Lasso regression is basically Shrinkage(regularization) techniques, which uses different parameters and values to shrink or penalize the coefficients. When we fit a model, we are asking it to learn a set of coefficients that best fit over the training distribution as well as hope to generalize on test data points as well. Learning those coefficients can be done in various ways and multiple techniques are there to reduce the error in coefficients such as LMS(Least mean Squared), RSS(Residual Sum of Squares). Now suppose we are training a model using either LMS or RSS, then ridge regression makes use of the extra term which penalizes the results of LMS or RSS towards zero. So, simply written, Final error to be corrected = RSS + Ridge Term [OR] []1 where Beta1, Beta2 ... are the coefficients for X1, X2... and so on.  Ridge term includes the alpha term, which is nothing but the penalty or the tuning parameter. The whole ridge term is sometimes called the shrinkage penalty term too. If we fit the data very well, the RSS value is very low. But the second term is close to zero only when B1, B2...Bn values are small. If these are small, then the corresponding X1, X2....Xn values will be small. Thus the impact of Xi term on Y(the output variable) will be less significant compared to some Bj for Xj whose value is large enough. The alpha term acts as the control parameter, which determines, how much significance should be given to Xi for the Bi coefficient. If Alpha is close to zero, the Ridge term itself is very small and thus the final error is based on RSS alone. If Alpha is too large, the impact of shrinkage grows and the coefficients B1, B2 ... Bn tends to zero.  Choosing the right value helps the model learn the right features and better generalize the coefficients. One of the methods that help in choosing the right value is Cross-validation.
Question: For some reason, Cython is returning 0 on a math expression that should evaluate to 0.5: print(2 ** (-1)) # prints 0  Oddly enough, mix variables in and it'll work as expected: i = 1 print(2 ** (-i)) # prints 0.5  Vanilla CPython returns 0.5 for both cases. I'm compiling for 37m-x86_64-linux-gnu, and language_level is set to 3. What is this witchcraft?

Answer: It's because it's using C ints rather than Python integers so it matches C behaviour rather than Python behaviour. I'm relatively sure this used to be documented as a limitation somewhere but I can't find it now. If you want to report it as a bug then go to https://github.com/cython/cython/issues, but I suspect this is a deliberate trade-off of speed for compatibility. The code gets translated to __Pyx_pow_long(2, -1L)  where __Pyx_pow_long is a function of type static CYTHON_INLINE long __Pyx_pow_long(long b, long e).  The easiest way to fix it is to change one/both of the numbers to be a floating point number  print(2. ** (-1))   As a general comment on the design choice: people from the C world generally expect int operator int to return an int, and this option will be fastest. Python had tried to do this in the past with the Python 2 division behaviour (but inconsistently - power always returned a floating point number). Cython generally tries to follow Python behaviour. However, a lot of people are using it for speed so they also try to fall back to quick, C-like operations especially when people specify types (since those people want speed). I think what's happened here is that it's been able to infer the types automatically, and so defaulted to C behaviour. I suspect ideally it should distinguish between specified types and types that it's inferred. However, it's also probably too late to start changing that.
Question: According to asyncio synchronization primitives, there are synchronization methods.  I am getting confused about why we need synchronization in asyncio? I mean, asyncio is asynchronous. Is it meaningful to add something synchronous in asynchronization? 

Answer: Synchronization primitives don't make your code synchronous, they make coroutines in your code synchronized. Few examples:  You may want to start/continue some coroutine only when another coroutine allows it (asyncio.Event) You may want some part of your code to be executed only by single coroutine at the same time and other to wait for their turn (asyncio.Lock) You may want some part of your code to be executed only by limited number on coroutines at the same time (asyncio.Semaphore)  Take a look at a practical example of using asyncio.Semaphore.
Question: I have a python script that I'm trying to execute but i'm not sure how it's meant to be executed. I don't program in Python so i'm unfamiliar with the language. Here is the link to the script i'm trying to use. Also, a link the configuration it's using if you wish to see it. All it seems to do for what's relevant here, however, is set my path which I know is correct since other scripts (not linked here) work as expected with the configuration in that file.  Having a look at the script, I believe that the script is meant to be ran with the command line arguments: view, new, init. Thus, I ran the following in my terminal $ lectures.py new  But I get the following traceback Traceback (most recent call last): File "/usr/bin/lectures.py", line 156, in <module> lectures = Lectures(Path.cwd()) File "/usr/bin/lectures.py", line 60, in __init__ self.root = course.path AttributeError: 'PosixPath' object has no attribute 'path'  Furthermore, my python version $ python --version Python 3.8.1  EDIT: I wanted to add the reference as well for what I am trying to follow 

Answer: Going through your code, I think you might mean: self.root = course  at that line. Path.cwd() returns:  ... the current working directory, that is, the directory from where you run the script.  that is, either a WindowsPath() or a PosixPath object. I believe it is PosixPath for you, and you can verify with: import os print(os.name) # posix -> Linux # nt -> Windows  This has no attribute path, and this is what your Interpreter tells you.
Question: I'm learning to use docker SDK. I understand that containers need to be removed after run, otherwise requiring pruning later on. I'm see that there are two boolean flags in client.containers.run:   auto_remove (bool) – enable auto-removal of the container on daemon side when the container’s process exits. remove (bool) – Remove the container when it has finished running. Default: False   What's the difference? If auto-remove is on daemon side, which side is remove on? Angel? Which side should I join??  ref: https://docker-py.readthedocs.io/en/stable/containers.html

Answer: It's in fact exactly that: AutoRemove is one of the parameters to the "create a container" Docker API call, but the remove option signals the client library to remove the container after it exits. Setting auto_remove: True is probably more robust (the container will still clean itself up if the coordinator process crashes) but if the container fails with that option set then container.run() won't return its stderr. If you set detach: True to get back a Container object, then you can't use remove: True (it gets converted to auto_remove: True) but your code can container.remove() it after it's exited.
Question: e.g. from queue import Queue q: Queue = Queue() q.put("abc")  This is okay. Now I want to specify the types that go into the queue. from queue import Queue q: Queue[str] = Queue() q.put("abc")  This gets "TypeError: 'type' object is not subscriptable"

Answer: This isn't currently supported by the typing functionality. See this discussion of Python core developers: https://bugs.python.org/issue33315 It also suggests a current workaround, to put the annotation in quotes: q: "Queue[str]" = Queue()  The workaround is that an annotation can be any type (no pun intended). A string is a perfectly acceptable type, but if you want to use non-string annotations, then they have to "behave" properly. You can annotate that something is a Queue because, again, an annotation can have any type. But the Queue class itself isn't subscriptable, so Queue[int] doesn't work.
Question: I have an uneven colormap and I want the 0 to be white. All negative colors have to be bluish and all positive colors have to be reddish. My current attempt displays the 0 bluish and the 0.7 white. Is there any way to set the 0 to white? import numpy as np import matplotlib.colors as colors from matplotlib import pyplot as m bounds_min = np.arange(-2, 0, 0.1) bounds_max = np.arange(0, 4.1, 0.1) bounds = np.concatenate((bounds_min, bounds_max), axis=None) norm = colors.BoundaryNorm(boundaries=bounds, ncolors=256) # I found this on the internet and thought this would solve my problem. But it doesn't... m.pcolormesh(xx, yy, interpolated_grid_values, norm=norm, cmap='RdBu_r') 

Answer: The other answer makes it a little more complicated than it needs to be. In order to have the middle point of the colormap at 0, use a DivergingNorm with vcenter=0. import numpy as np import matplotlib.pyplot as plt from matplotlib.colors import DivergingNorm x, y = np.meshgrid(np.linspace(0,50,51), np.linspace(0,50,51)) z = np.linspace(-2,4,50*50).reshape(50,50) norm = DivergingNorm(vmin=z.min(), vcenter=0, vmax=z.max()) pc = plt.pcolormesh(x,y,z, norm=norm, cmap="RdBu_r") plt.colorbar(pc) plt.show()   Note: From matplotlib 3.2 onwards DivergingNorm will be renamed to TwoSlopeNorm
Question: I'm trying to add a label or hover column to the points in a scatter plot, but to no avail. For use as example data: import pandas as pd import holoviews as hv import hvplot.pandas df = pd.read_csv('http://assets.holoviews.org/macro.csv', '\t') df.query("year == 1966").hvplot.scatter(x="gdp", y="unem")  results in the picture below. If I hover over this item I cannot see, which country is represented by the dot (which makes it rather useless). I could use the additional keyword color="country"in the scatterplot call. This would lead to an additional legend (which could be turned off) and the value country: countryname is added to the hover field.  Is there an option that just adds the column to my hover without adding a legend and changing the color?

Answer: You can use keyword hover_cols to add additional columns to your hover. Documentation: https://hvplot.holoviz.org/user_guide/Customization.html  hover_cols (default=[]): list or str  Additional columns to add to the hover tool or 'all' which will includes all columns (including indexes if use_index is True).   So in your example if you would like to add specific columns to your hover: # use keyword hover_cols to add additional columns when hovering df.hvplot.scatter(x="gdp", y="unem", hover_cols=['country', 'year'])   Or if you want to include all additional columns to your hover: df.hvplot.scatter(x="gdp", y="unem", hover_cols='all')   Or if you want to include all columns to hover, but not the index: df.hvplot.scatter(x="gdp", y="unem", hover_cols='all', use_index=False)
Question: I have a use case where i have to find the consecutive rgb pixel color count of each frame of live video after searching i found a piece of code which does the same thing but performance wise it take around ~ 3 sec to give me output but in my case i have to do this calculation as fast as possible may be 25 frames in 1 seconds. Can someone help me to figure out how to do this by refactoring the below code from PIL import Image import timeit starttime = timeit.default_timer() with Image.open("netflix.png") as image: color_count = {} width, height = image.size print(width,height) rgb_image = image.convert('RGB') for x in range(width): for y in range(height): rgb = rgb_image.getpixel((x, y)) if rgb in color_count: color_count[rgb] += 1 else: color_count[rgb] = 1 print('Pixel Count per Unique Color:') print('-' * 30) print(len(color_count.items())) print("The time difference is :", timeit.default_timer() - starttime)  output: Pixel Count per Unique Color: 130869 The time difference is : 3.9660612

Answer: You need to use Numpy, or OpenCV, for fast image processing in Python. I made a 9-colour version of Paddington:  from PIL import Image import numpy as np # Open Paddington and make sure he is RGB - not palette im = Image.open('paddington.png').convert('RGB') # Make into Numpy array na = np.array(im) # Arrange all pixels into a tall column of 3 RGB values and find unique rows (colours) colours, counts = np.unique(na.reshape(-1,3), axis=0, return_counts=1) print(colours) print(counts)  Results [[ 14 48 84] [ 19 21 30] [ 33 108 163] [ 33 152 190] [ 72 58 58] [ 96 154 210] [180 89 64] [205 210 200] [208 151 99]] [20389 40269 12820 1488 17185 25371 17050 16396 9032]  That means there are 20,389 pixels of RGB(14,48,84), and so on. That takes 125ms on my Mac for a 400x400 image, which will give you 8 fps, so you better have at least 4 CPU cores and use all of them to get 25+ fps.  Update I think you can actually go significantly faster than this. If you take the dot-product of each of the pixels with [1,256,65536], you will get a single 24-bit number for each pixel, rather than 3 8-bit numbers. It is then a lot faster to find the unique values. That looks like this: # Open Paddington and make sure he is RGB - not palette im = Image.open('paddington.png').convert('RGB') # Make into Numpy array na = np.array(im) # Make a single 24-bit number for each pixel f = np.dot(na.astype(np.uint32),[1,256,65536]) nColours = len(np.unique(f)) # prints 9  That takes 4ms rather than 125ms on my Mac :-)  Keywords: Python, Numpy, PIL/Pillow, image processing, count unique colours, count colors.
Question: The scipy.stats library has functions to find the mean and median of a fitted distribution but not mode.  If I have the parameters of a distribution after fitting to data, how can I find the mode of the fitted distribution? 

Answer: If I don't get your wrong, you want to find the mode of fitted distributions instead of mode of a given data. Basically, we can do it with following 3 steps. Step 1: generate a dataset from a distribution from scipy import stats from scipy.optimize import minimize # generate a norm data with 0 mean and 1 variance data = stats.norm.rvs(loc= 0,scale = 1,size = 100) data[0:5]  Output:  array([1.76405235, 0.40015721, 0.97873798, 2.2408932 , 1.86755799])  Step 2: fit the parameters # fit the parameters of norm distribution params = stats.norm.fit(data) params  Output:  (0.059808015534485, 1.0078822447165796)  Note that there are 2 parameters for stats.norm, i.e. loc and scale. For different dist in scipy.stats, the parameters are different. I think it's convenient to store parameter in a tuple and then unpack it in the next step. Step 3: get the mode(maximum of your density function) of fitted distribution # continuous case def your_density(x): return -stats.norm.pdf(x,*paras) minimize(your_density,0).x  Output:  0.05980794  Note that a norm distribution has mode equals to mean. It's a coincidence in this example.  One more thing is that scipy treats continuous dist and discrete dist different(they have different father classes), you can do the same thing with following code on discrete dists. ## discrete dist, example for poisson x = np.arange(0,100) # the range of x should be specificied x[stats.poisson.pmf(x,mu = 2).argmax()] # find the x value to maximize pmf  Out:  1  You can it try with your own data and distributions!
Question: I'm trying to compute the AUC score for a multiclass problem using the sklearn's roc_auc_score() function. I have prediction matrix of shape [n_samples,n_classes] and a ground truth vector of shape [n_samples], named np_pred and np_label respectively. What I'm trying to achieve is the set of AUC scores, one for each classes that I have. To do so I would like to use the average parameter option None and multi_class parameter set to "ovr", but if I run  roc_auc_score(y_score=np_pred, y_true=np_label, multi_class="ovr",average=None)  I get back ValueError: average must be one of ('macro', 'weighted') for multiclass problems  This error is expected from the sklearn function in the case of the multiclass; but if you take a look at the roc_auc_score function source code, you can see that if the multi_class parameter is set to "ovr", and the average is one of the accepted one, the multiClass case is treated as a multiLabel one and the internal multiLabel function accepts None as average parameter. So, by looking at the code, it seems that I should be able to execute a multiclass with a None average in a One vs Rest case but the ifs in the source code do not allow such combination. Am I wrong? In case I'm wrong, from a theoretical point of view should I fake a multilabel case just to have the different AUCs for the different classes or should I write my own function that cycles the different classes and outputs the AUCs? Thanks

Answer: As you already know, right now sklearn multiclass ROC AUC only handles the macro and weighted averages. But it can be implemented as it can then individually return the scores for each class. Theoretically speaking, you could implement OVR and calculate per-class roc_auc_score, as: roc = {label: [] for label in multi_class_series.unique()} for label in multi_class_series.unique(): selected_classifier.fit(train_set_dataframe, train_class == label) predictions_proba = selected_classifier.predict_proba(test_set_dataframe) roc[label] += roc_auc_score(test_class, predictions_proba[:,1])