eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I try to write a simple middleware for FastAPI peeking into response bodies. In this example I just log the body content: app = FastAPI() @app.middleware("http") async def log_request(request, call_next): logger.info(f'{request.method} {request.url}') response = await call_next(request) logger.info(f'Status code: {response.status_code}') async for line in response.body_iterator: logger.info(f' {line}') return response  However it looks like I "consume" the body this way, resulting in this exception:  ... File ".../python3.7/site-packages/starlette/middleware/base.py", line 26, in __call__ await response(scope, receive, send) File ".../python3.7/site-packages/starlette/responses.py", line 201, in __call__ await send({"type": "http.response.body", "body": b"", "more_body": False}) File ".../python3.7/site-packages/starlette/middleware/errors.py", line 156, in _send await send(message) File ".../python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 515, in send raise RuntimeError("Response content shorter than Content-Length") RuntimeError: Response content shorter than Content-Length  Trying to look into the response object I couldn't see any other way to read its content. What is the correct way to do it?

Answer: I had a similar need in a FastAPI middleware and although not ideal here's what we ended up with: app = FastAPI() @app.middleware("http") async def log_request(request, call_next): logger.info(f'{request.method} {request.url}') response = await call_next(request) logger.info(f'Status code: {response.status_code}') body = b"" async for chunk in response.body_iterator: body += chunk # do something with body ... return Response( content=body, status_code=response.status_code, headers=dict(response.headers), media_type=response.media_type )  Be warned that such an implementation is problematic with responses streaming a body that would not fit in your server RAM (imagine a response of 100GB). Depending on what your application does, you will rule if it is an issue or not.  In the case where some of your endpoints produce large responses, you might want to avoid using a middleware and instead implement a custom ApiRoute. This custom ApiRoute would have the same issue with consuming the body, but you can limit it's usage to a particular endpoints. Learn more at https://fastapi.tiangolo.com/advanced/custom-request-and-route/
Question: I'm trying to use plot_confusion_matrix, from sklearn.metrics import confusion_matrix y_true = [1, 1, 0, 1] y_pred = [1, 1, 0, 0] confusion_matrix(y_true, y_pred)  Output: array([[1, 0], [1, 2]])  Now, while using the followings; using 'classes' or without 'classes' from sklearn.metrics import plot_confusion_matrix plot_confusion_matrix(y_true, y_pred, classes=[0,1], title='Confusion matrix, without normalization')  or plot_confusion_matrix(y_true, y_pred, title='Confusion matrix, without normalization')  I expect to get similar output like this except the numbers inside,  Plotting simple diagram, it should not require the estimator. Using mlxtend.plotting,  from mlxtend.plotting import plot_confusion_matrix import matplotlib.pyplot as plt import numpy as np binary1 = np.array([[4, 1], [1, 2]]) fig, ax = plot_confusion_matrix(conf_mat=binary1) plt.show()  It provides same output. Based on this it requires a classifier, disp = plot_confusion_matrix(classifier, X_test, y_test, display_labels=class_names, cmap=plt.cm.Blues, normalize=normalize)  Can I plot it without a classifier?

Answer: plot_confusion_matrix expects a trained classifier. If you look at the source code, what it does is perform the prediction to generate y_pred for you: y_pred = estimator.predict(X) cm = confusion_matrix(y_true, y_pred, sample_weight=sample_weight, labels=labels, normalize=normalize)  So in order to plot the confusion matrix without specifying a classifier, you'll have to go with some other tool, or do it yourself. A simple option is to use seaborn: import seaborn as sns cm = confusion_matrix(y_true, y_pred) f = sns.heatmap(cm, annot=True)
Question: Let's say we have a complex API function, imported from some library. def complex_api_function( number, <lots of positional arguments>, <lots of keyword arguments>): '''really long docstring''' # lots of code  I want to write a simple wrapper around that function to make a tiny change. For example, it should be possible to pass the first argument as a string. How to document this? I considered the following options: Option 1: def my_complex_api_function(number_or_str, *args, **kwargs): ''' Do something complex. Like `complex_api_function`, but first argument can be a string. Parameters ---------- number_or_str : int or float or str Can be a number or a string that can be interpreted as a float. <copy paste description from complex_api_function docstring> *args Positional arguments passed to `complex_api_function`. **kwargs Keyword arguments passed to `complex_api_function`. Returns ------- <copy paste from complex_api_function docstring> Examples -------- <example where first argument is a string, e.g. '-5.0'> ''' return complex_api_function(float(number_or_str), *args, **kwargs)  Disadvantage: User has to look at the docs of complex_api_function to get information about *args and **kwargs. Needs adjustment when the copy pasted sections from complex_api_function change. Option 2: Copy and paste complex_api_function's signature (instead of using *args and **kwargs) and its docstring. Make a tiny change to the docstring that mentions that the first argument can also be a string. Add an example. Disadvantage: verbose, has to be changed when complex_api_function changes. Option 3: Decorate my_complex_api_function with functools.wraps(complex_api_function). Disadvantage: There's no information that number can also be a string.  I'm looking for an answer that does not hinge on the details of what changes in my_complex_api_function. The procedure should work for any tiny adjustment to the original complex_api_function.

Answer: I'd recommend something like the following: def my_complex_api_function(number_or_str, *args, **kwargs): """This function is a light wrapper to `complex_api_function`. It allows you to pass a string or a number, whereas `complex_api_function` requires a number. See :ref:`complex_api_function` for more details. :param number_or_str: number or str to convert to a number and pass to `complex_api_function`. :param args: Arguments to pass to `complex_api_function` :param kwargs: Keyword arguments to pass to `complex_api_function` :return: Output of `complex_api_function`, called with passed parameters """  This is clear and concise. But please also remember that, if using a documentation system like sphinx, to link the functions with :ref:`bob` or similar.
Question: When I stop typing in Spyder 4, all occurrences of the last word are automatically highlighted after about two seconds. Is it a bug or can I disable it? I use Spyder on Ubuntu 18.04. 

Answer: I found the setting: Tools > Preferences > Editor > Display > Highlight occurrences after
Question: There are questions that addresses the same error TypeError: Timestamp subtraction must have the same timezones or no timezones but none faces the same issue as this one. I have 2 UTC Timestamps that throw that error when substracted. print(date, type(date), date.tzinfo) >>> 2020-07-17 00:00:00+00:00 <class 'pandas._libs.tslibs.timestamps.Timestamp'> UTC print(date2, type(date2), date2.tzinfo) >>> 2020-04-06 00:00:00.000000001+00:00 <class 'pandas._libs.tslibs.timestamps.Timestamp'> UTC date - date2 >>> TypeError: Timestamp subtraction must have the same timezones or no timezones  Edit: I'm using Python 3.6.9 and Pandas 1.0.1 

Answer: After checking the timezone types: type(date.tzinfo) gives <class 'datetime.timezone'> and type(date2.tzinfo) gives <class 'pytz.UTC'> so acording of pandas source code they are not considered equal even even if they are both UTC. So the solution was to make them have the same tzinfo type (either pytz or datitme.timezone) This is an open issue in Github: https://github.com/pandas-dev/pandas/issues/32619
Question: I'm trying to create a line chart with point markers in Altair. I'm using the multi-series line chart example from Altair's documentation and trying to combine it with the line chart with stroked point markers example from Vega-Lite's documentation.  Where I'm confused is how to handle the 'mark_line' argument. From the Vega example, I need use "point" and then set "filled" to False.   "mark": { "type": "line", "point": { "filled": false, "fill": "white" } },  How would I apply that in Altair? I figured out that setting 'point' to 'True' or '{}' added a point marker, but confused on how to get the fill to work.  source = data.stocks() alt.Chart(source).mark_line( point=True ).encode( x='date', y='price', color='symbol' ) 

Answer: You can always pass a raw vega-lite dict to any property in Altair: source = data.stocks() alt.Chart(source).mark_line( point={ "filled": False, "fill": "white" } ).encode( x='date', y='price', color='symbol' )  or you can check the docstring of mark_line() and see that it expects point to be an OverlayMarkDef() and use the Python wrappers: alt.Chart(source).mark_line( point=alt.OverlayMarkDef(filled=False, fill='white') ).encode( x='date', y='price', color='symbol' )
Question: Is there any rolling "any" function in a pandas.DataFrame? Or is there any other way to aggregate boolean values in a rolling function? Consider: import pandas as pd import numpy as np s = pd.Series([True, True, False, True, False, False, False, True]) # this works but I don't think it is clear enough - I am not # interested in the sum but a logical or! s.rolling(2).sum() > 0 # What I would like to have: s.rolling(2).any() # AttributeError: 'Rolling' object has no attribute 'any' s.rolling(2).agg(np.any) # Same error! AttributeError: 'Rolling' object has no attribute 'any'  So which functions can I use when aggregating booleans? (if numpy.any does not work) The rolling documentation at https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.DataFrame.rolling.html states that "a Window or Rolling sub-classed for the particular operation" is returned, which doesn't really help.

Answer: This method is not implemented, close, what you need is use Rolling.apply: s = s.rolling(2).apply(lambda x: x.any(), raw=False) print (s) 0 NaN 1 1.0 2 1.0 3 1.0 4 1.0 5 0.0 6 0.0 7 1.0 dtype: float64 s = s.rolling(2).apply(lambda x: x.any(), raw=False).fillna(0).astype(bool) print (s) 0 False 1 True 2 True 3 True 4 True 5 False 6 False 7 True dtype: bool  Better here is use strides - generate numpy 2d arrays and processing later: s = pd.Series([True, True, False, True, False, False, False, True]) def rolling_window(a, window): shape = a.shape[:-1] + (a.shape[-1] - window + 1, window) strides = a.strides + (a.strides[-1],) return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides) a = rolling_window(s.to_numpy(), 2) print (a) [[ True True] [ True False] [False True] [ True False] [False False] [False False] [False True]] print (np.any(a, axis=1)) [ True True True True False False True]  Here first NaNs pandas values are omitted, you can add first values for processing, here Falses: n = 2 x = np.concatenate([[False] * (n), s]) def rolling_window(a, window): shape = a.shape[:-1] + (a.shape[-1] - window + 1, window) strides = a.strides + (a.strides[-1],) return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides) a = rolling_window(x, n) print (a) [[False False] [False True] [ True True] [ True False] [False True] [ True False] [False False] [False False] [False True]] print (np.any(a, axis=1)) [False True True True True True False False True]
Question: I want to write my test using pytest and to be able to run them (individually) in IntelliJ. I have pytest installed, along with (obviously) Python plugin for the IDE. My test file (tests/test_main.py) looks like that: from app.main import sum_numbers def test_sum_numbers(): assert sum_numbers(1, 2) == 3 assert sum_numbers(10, 20) == 30 assert sum_numbers(1, 2, -3) == 0  Running pytest from terminal works perfectly. But IntelliJ seems to treat this as a regular file, there are no "run" icons before the declarations. Also, running the file from the IDE does nothing, as it is not treated as a test file. What can I do?

Answer: Here what you need to do: 1) Remove all the existing run configurations for your test file. 2) Make sure that Preferences | Tools | Python Integrated Tools | Default rest runner is set to pytest. After that, you should see the run icon next to your test functions and right-clicking the test file should suggest running the pytest. See the docs for more details.
Question: I'm having a conundrum with the Python Click library when parsing some CLI options. I would like an option to act as a flag by itself, but optionally accept string values. E.g.:  $ myscript ⇒ option = False $ myscript -o ⇒ option = True $ myscript -o foobar ⇒ option = Foobar  Additionally, I would like the option to be "eager" (e.g. in "Click" terms abort execution after a callback), but this can be ignored for now.  When I define my arguments like this: @click.command() @click... @click.option("-o", "option", is_flag=True, default=False) def myscript(..., option):  I achieve example 1 and 2, but 3 is naturally impossible because the flag detects present/not present only.  When I define my arguments like this: @click.command() @click... @click.option("-o", "--option", default="") # Let's assume I will cast empty string to False def myscript(..., option):  I achieve 1 and 3, but 2 will fail with an Error: -c option requires an argument.  This does not seems like an out-of-this world scenario, but I can't seem to be able to achieve this or find examples that behave like this. How can I define an @click.option that gets parsed like:  False when not set True when set but without value str when set with value 

Answer: One way that I have managed to achieve this behaviour was by actually using arguments as below. I'll post this as a workaround, while I try to see if it could be done with an option, and I'll update my post accordingly @click.command(context_settings={"ignore_unknown_options": True}) @click.argument("options", nargs=-1) def myscript(options): option = False if options is (): option = False if '-o' in options or '--option' in options: option = True if len(options) > 1: option = options[1] print(option)  Later Edit Using an option, I have managed to achieve this by adding an argument to the command definition. @click.command() @click.option('-o', '--option', is_flag=True, default=False, is_eager=True) @click.argument('value', nargs=-1) def myscript(option, value): if option and value != (): option = value[0] print(option)  The nargs can be removed if you only expect at most one argument to follow, and can be treated as not required. @click.command() @click.option('-o', '--option', is_flag=True, default=False, is_eager=True) @click.argument('value', required=False) def myscript(option, value=None): if option and value is not None: option = value print(option)  It might also be possible by putting together a context generator and storing some state, but that seems the least desirable solution, since you would be relying on the context storing your state.
Question: I want to convert a RGB image into CMYK. This is my code; the first problem is when I divide each pixel by 255, the value closes to zero, so the resulting image is approximately black! The second problem is that I don't know how to convert the one-channel resultant image to 4 channels. Of course, I'm not sure the made CMYK in the following code is correct. Thank you for your attention    import cv2 import numpy as np import time img = cv2.imread('image/dr_trump.jpg') B = img[:, :, 0] G = img[:, :, 1] R = img[:, :, 2] B_ = np.copy(B) G_ = np.copy(G) R_ = np.copy(R) K = np.zeros_like(B) C = np.zeros_like(B) M = np.zeros_like(B) Y = np.zeros_like(B) ts = time.time() for i in range(B.shape[0]): for j in range(B.shape[1]): B_[i, j] = B[i, j]/255 G_[i, j] = G[i, j]/255 R_[i, j] = R[i, j]/255 K[i, j] = 1 - max(B_[i, j], G_[i, j], R_[i, j]) if (B_[i, j] == 0) and (G_[i, j] == 0) and (R_[i, j] == 0): # black C[i, j] = 0 M[i, j] = 0 Y[i, j] = 0 else: C[i, j] = (1 - R_[i, j] - K[i, j])/float((1 - K[i, j])) M[i, j] = (1 - G_[i, j] - K[i, j])/float((1 - K[i, j])) Y[i, j] = (1 - B_[i, j] - K[i, j])/float((1 - K[i, j])) CMYK = C + M + Y + K t = (time.time() -ts) print("Loop: {:} ms".format(t*1000)) cv2.imshow('CMYK by loop',CMYK) cv2.waitKey(0) cv2.destroyAllWindows() 

Answer: You can let PIL/Pillow do it for you like this: from PIL import Image # Open image, convert to CMYK and save as TIF Image.open('drtrump.jpg').convert('CMYK').save('result.tif')  If I use IPython, I can time loading, converting and saving that at 13ms in toto like this: %timeit Image.open('drtrump.jpg').convert('CMYK').save('PIL.tif') 13.6 ms ± 627 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)   If you want to do it yourself by implementing your formula, you would be better off using vectorised Numpy rather than for loops. This takes 35ms. #!/usr/bin/env python3 import cv2 import numpy as np # Load image bgr = cv2.imread('drtrump.jpg') # Make float and divide by 255 to give BGRdash bgrdash = bgr.astype(np.float)/255. # Calculate K as (1 - whatever is biggest out of Rdash, Gdash, Bdash) K = 1 - np.max(bgrdash, axis=2) # Calculate C C = (1-bgrdash[...,2] - K)/(1-K) # Calculate M M = (1-bgrdash[...,1] - K)/(1-K) # Calculate Y Y = (1-bgrdash[...,0] - K)/(1-K) # Combine 4 channels into single image and re-scale back up to uint8 CMYK = (np.dstack((C,M,Y,K)) * 255).astype(np.uint8)   If you want to check your results, you need to be aware of a few things. Not all image formats can save CMYK, that's why I saved as TIFF. Secondly, your formula leaves all your values as floats in the range 0..1, so you probably want scale back up by multiplying by 255 and converting to uint8. Finally, you can be assured of what the correct result is by simply using ImageMagick in the Terminal: magick drtrump.jpg -colorspace CMYK result.tif
Question: can someone explain why this simple code wont execute the facecolor command while setting the axis off?  fig = plt.figure(1) ax = fig.add_subplot(211, facecolor=(0,0,0), aspect='equal') ax.scatter(np.random.random(10000), np.random.random(10000), c="gray", s=0.25) ax.axes.set_axis_off()  Thanks in advance! 

Answer: The background patch is part of the axes. So if the axes is turned off, so will the background patch.  Some options: Re-add the background patch ax = fig.add_subplot(211, facecolor=(0,0,0), aspect='equal') ax.set_axis_off() ax.add_artist(ax.patch) ax.patch.set_zorder(-1)  Create new patch ax = fig.add_subplot(211, facecolor=(0,0,0), aspect='equal') ax.set_axis_off() ax.add_patch(plt.Rectangle((0,0), 1, 1, facecolor=(0,0,0), transform=ax.transAxes, zorder=-1))  Turn axis spines and ticks invisible ...but keep the axis on. ax = fig.add_subplot(211, facecolor=(0,0,0), aspect='equal') for spine in ax.spines.values(): spine.set_visible(False) ax.tick_params(bottom=False, labelbottom=False, left=False, labelleft=False)
Question: An n-dimensional array has 2n sides (a 1-dimensional array has 2 endpoints; a 2-dimensional array has 4 sides or edges; a 3-dimensional array has 6 2-dimensional faces; a 4-dimensional array has 8 sides; etc.). This is analogous to what happens with abstract n-dimensional cubes. I want to check if all sides of an n-dimensional array are composed by only zeros. Here are three examples of arrays whose sides are composed by zeros: # 1D np.array([0,1,2,3,0]) # 2D np.array([[0, 0, 0, 0], [0, 1, 0, 0], [0, 2, 3, 0], [0, 0, 1, 0], [0, 0, 0, 0]]) # 3D np.array([[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]], [[0, 0, 0, 0], [0, 1, 2, 0], [0, 0, 0, 0]], [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]])  How can I check if all sides of a multidimensional numpy array are arrays of zeros? For example, with a simple 2-dimensional array I can do this: x = np.random.rand(5, 5) assert np.sum(x[0:, 0]) == 0 assert np.sum(x[0, 0:]) == 0 assert np.sum(x[0:, -1]) == 0 assert np.sum(x[-1, 0:]) == 0  While this approach works for 2D cases, it does not generalize to higher dimensions. I wonder if there is some clever numpy trick I can use here to make it efficient and also more maintainable.

Answer: Here's how you can do it: assert(all(np.all(np.take(x, index, axis=axis) == 0) for axis in range(x.ndim) for index in (0, -1)))  np.take does the same thing as "fancy" indexing.
Question: I would like to create a read-only reference to a NumPy array. Is this a correct way to make b a read-only reference to a (a is any NumPy array)? def get_readonly_view(a): b = a.view() b.flags.writeable = False return b  Specifically, I would like to ensure the above does not 'copy' the contents of a? (I tried testing this with np.shares_memory and it does return True. But I am not sure if that is a correct test.) In addition I wonder if get_readonly_view is already implemented in NumPy?  Update. It has been suggested to turn the array to a class property to make it read-only. I think this does not work: import numpy as np class Foo: def __init__(self): self._a = np.arange(15).reshape((3, 5)) @property def a(self): return self._a def bar(self): print(self._a)  But the client can change contents of _a: >> baz = Foo() >> baz.bar() [[ 0 1 2 3 4] [ 5 6 7 8 9] [10 11 12 13 14]] >> baz.a[1, 2] = 10 >> baz.bar() [[ 0 1 2 3 4] [ 5 6 10 8 9] [10 11 12 13 14]]  while I would like baz.a[1, 2] = 10 to raise an exception. 

Answer: Your approach seems to be the suggested way of creating a read-only view. In particular, arr.view() (which can also be written as a slicing arr[:]) will create a reference to arr, while modifying the writeable flag is the suggested way of making a NumPy array read-only. The documentation also provides some additional information on the inheritance of the writeable property:  The data area can be written to. Setting this to False locks the data, making it read-only. A view (slice, etc.) inherits WRITEABLE from its base array at creation time, but a view of a writeable array may be subsequently locked while the base array remains writeable. (The opposite is not true, in that a view of a locked array may not be made writeable. However, currently, locking a base object does not lock any views that already reference it, so under that circumstance it is possible to alter the contents of a locked array via a previously created writeable view onto it.) Attempting to change a non-writeable array raises a RuntimeError exception.  Just to reiterate and inspect what is going on: import numpy as np def get_readonly_view(arr): result = arr.view() result.flags.writeable = False return result a = np.zeros((2, 2)) b = get_readonly_view(a) print(a.flags) # C_CONTIGUOUS : True # F_CONTIGUOUS : False # OWNDATA : True # WRITEABLE : True # ALIGNED : True # WRITEBACKIFCOPY : False # UPDATEIFCOPY : False print(b.flags) # C_CONTIGUOUS : True # F_CONTIGUOUS : False # OWNDATA : False # WRITEABLE : False # ALIGNED : True # WRITEBACKIFCOPY : False # UPDATEIFCOPY : False print(a.base) # None print(b.base) # [[0. 0.] # [0. 0.]] a[1, 1] = 1.0 # ...works b[0, 0] = 1.0 # raises ValueError: assignment destination is read-only