eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am trying to export the forecasts I made with Pytorch models in darts library to some exchangeable format like XLS or CSV. Is it somehow possible to export the predictions from the Timeseries class? How can I specify output format?

Answer: Forecasts in Darts are nothing but regular TimeSeries objects and a TimeSeries is internally represented as a pandas.DataFrame. You can access this DataFrame using series.pd_dataframe() (to get a copy of the DataFrame) or series._df directly if you want to avoid copying the DataFrame to save it. Be careful in the latter case however, as modifying the DataFrame in place will break the TimeSeries immutability. You can then use any method you'd like to save pandas dataframes, e.g. pandas.DataFrame.to_csv() or pandas.DataFrame.to_pickle(). You can have a look at this article on Medium to see a comparison of a couple different formats' performances for saving and loading dataframes.
Question: I have a 4-d Pytorch tensor that I would like to save to disk as a .jpg My tensor is the following size: print(image_tensor.size()) >>>torch.Size([1, 3, 400, 711])  I can view the entire tensor as one image within my IDE: ax1.imshow(im_convert(image_tensor))  Since I am able to view the entire tensor as one image, I am assuming there is a way to also save it as such. However, when I try to save the image, it looks like it only saves the blue color channel. I would like to save the entire tensor as a single image. img1 = image_tensor[0] save_image(img1, 'img1.jpg') 

Answer: In PyTorch this snippet is working and saving the image: from torchvision.utils import save_image import torch import torchvision tensor= torch.rand(2, 3, 400, 711) img1 = tensor[0] save_image(img1, 'img1.png')  Before saving the image can you check the shape of the img1 in any case something happened.
Question: code: from plotly.offline import init_notebook_mode, iplot, iplot_mpl def plot_train_test(train, test, date_split): data = [Candlestick(x=train.index, open=train['open'], high=train['high'], low=train['low'], close=train['close'],name='train'), Candlestick(x=test.index, open=test['open'], high=test['high'], low=test['low'], close=test['close'],name='test') ] layout = { 'shapes': [ {'x0': date_split, 'x1': date_split, 'y0': 0, 'y1': 1, 'xref': 'x', 'yref': 'paper', 'line': {'color': 'rgb(0,0,0)', 'width': 1}}], 'annotations': [{'x': date_split, 'y': 1.0, 'xref': 'x', 'yref': 'paper', 'showarrow': False, 'xanchor': 'left','text': ' test data'}, {'x': date_split, 'y': 1.0, 'xref': 'x', 'yref': 'paper', 'showarrow': False, 'xanchor': 'right', 'text': 'train data '}] } figure = Figure(data=data, layout=layout) iplot(figure)  The above code is ok.But now I want to 'volume' in this candlestick chart code: from plotly.offline import init_notebook_mode, iplot, iplot_mpl def plot_train_test(train, test, date_split): data = [Candlestick(x=train.index, open=train['open'], high=train['high'], low=train['low'], close=train['close'],volume=train['volume'],name='train'), Candlestick(x=test.index, open=test['open'], high=test['high'], low=test['low'],close=test['close'],volume=test['volume'],name='test')] layout = { 'shapes': [ {'x0': date_split, 'x1': date_split, 'y0': 0, 'y1': 1, 'xref': 'x', 'yref': 'paper', 'line': {'color': 'rgb(0,0,0)', 'width': 1}} ], 'annotations': [ {'x': date_split, 'y': 1.0, 'xref': 'x', 'yref': 'paper', 'showarrow': False, 'xanchor': 'left', 'text': ' test data'}, {'x': date_split, 'y': 1.0, 'xref': 'x', 'yref': 'paper', 'showarrow': False, 'xanchor': 'right', 'text': 'train data '} ] } figure = Figure(data=data, layout=layout) iplot(figure)  error:  ValueError: Invalid property specified for object of type plotly.graph_objs.Candlestick: 'volume' 

Answer: You haven't provided a complete code snippet with a data sample, so I'm going to have to suggest a solution that builds on an example here. In any case, you're getting that error message simply because go.Candlestick does not have a Volume attribute. And it might not seem so at first, but you can easily set up go.Candlestick as an individual trace, and then include an individual go.Bar() trace for Volumes using:  fig = make_subplots(specs=[[{"secondary_y": True}]]) fig.add_traces(go.Candlestick(...), secondary_y=True) fig.add_traces(go.Bar(...), secondary_y=False)  Plot:  Complete code: import plotly.graph_objects as go from plotly.subplots import make_subplots import pandas as pd # data df = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/finance-charts-apple.csv') # Create figure with secondary y-axis fig = make_subplots(specs=[[{"secondary_y": True}]]) # include candlestick with rangeselector fig.add_trace(go.Candlestick(x=df['Date'], open=df['AAPL.Open'], high=df['AAPL.High'], low=df['AAPL.Low'], close=df['AAPL.Close']), secondary_y=True) # include a go.Bar trace for volumes fig.add_trace(go.Bar(x=df['Date'], y=df['AAPL.Volume']), secondary_y=False) fig.layout.yaxis2.showgrid=False fig.show()
Question: %matplotlib inline corr = df.corr() f, ax = plt.subplots(figsize=(11, 9)) ax = sns.heatmap( corr, vmin=-1, vmax=1, center=0, cmap=sns.diverging_palette(20, 220, n=500), linewidths=.50, cbar_kws={"shrink": .7}, square=True ) ax.set_xticklabels( ax.get_xticklabels(), rotation=45, horizontalalignment='right' ); plt.show()  This code is does not give any plot display on azure data bricks, only displays <Figure size 1100x900 with 2 Axes>  whereas the same code worked fine and displayed a corr plot earlier, not sure whats going wrong here. I get the same output even when I try this. mask = np.triu(np.ones_like(corr, dtype=bool)) f, ax = plt.subplots(figsize=(11, 9)) cmap = sns.diverging_palette(20, 220, as_cmap=True) sns.heatmap(corr, mask=mask, cmap=cmap, vmax=0.3, center=0, square=True, linewidths=.1, cbar_kws={"shrink": .7}) plt.show() 

Answer: It looks like an issue with the matplotlib modules above 3.3.0.  To know the exact reason, I would suggest you to report here: https://github.com/matplotlib/matplotlib/issues  As per the test from our end, you will experience the following error message with the matplotlib modules above 3.3.0.  If you have installed matplotlib modules above 3.3.0, I would suggest you to use matplotlib modules below 3.2.2.  Once you have installed matplotlib==3.2.2, I'm able to successfully display the plot.
Question: import pandas as pd df = pd.DataFrame({"col1": ["a", "b", "c", ["a", "b"]]})  I have a dataframe like this, and I want to find the rows that contains list in that column. I tried value_counts() but it tooks so long and throws error at the end. Here is the error: TypeError Traceback (most recent call last) pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.map_locations() TypeError: unhashable type: 'list' Exception ignored in: 'pandas._libs.index.IndexEngine._call_map_locations' Traceback (most recent call last): File "pandas/_libs/hashtable_class_helper.pxi", line 1709, in pandas._libs.hashtable.PyObjectHashTable.map_locations TypeError: unhashable type: 'list' c 1 a 1 [a, b] 1 b 1 Name: col1, dtype: int64  For bigger dataframes this tooks forever. Here is how the desired output look like: col1 c 1 b 1 [a,b] 1 dtype: int64 

Answer: Lists are mutable, they cannot be compared, so you can neither count the values nor set them as index. You would need to convert to tuple or set (thanks @CameronRiddell) to be able to count: df['col1'].apply(lambda x: tuple(x) if isinstance(x, list) else x).value_counts()  Output: c 1 b 1 a 1 (a, b) 1 Name: col1, dtype: int64
Question: I have df= a 1 nan 3  I want some syntax for df["b"] = df["a"] or 5 to create a b 1 1 nan 5 3 3  Does pandas support something like this? BONUS: what about different default values per index/group/anything?

Answer: import pandas as pd import numpy as np df = pd.DataFrame({"a": [1, np.nan, 3]}) df["b"] = df["a"].fillna(5) print(df)    a b 0 1.0 1.0 1 NaN 5.0 2 3.0 3.0   digging through the doc gave the standard pandas solution. No need to go through numpy.
Question: I'm trying to solve a problem in Python/Pandas which I think is closely related to the longest path algorithm. The DataFrame I'm working with has the following structure: import numpy as np import pandas as pd data = { "cusID": ["001", "001", "001", "001", "001", "001", "002", "002", "002"], "start": ["A", "B", "C", "D", "A", "E", "B", "C", "D"], "end": ["B", "C", "D", "A", "E", "A", "C", "D", "E"] } df = pd.DataFrame(data) print(df) cusID start end 0 001 A B 1 001 B C 2 001 C D 3 001 D A 4 001 A E 5 001 E A 6 002 B C 7 002 C D 8 002 D E  For each customer, I want to find the longest sequence which does not contain A. For instance, for customer 001, the sequence can be viewed as follows: A -> B -> C -> D -> A -> E -> A where B -> C -> D is the longest sequence of size 3. The resulting DataFrame I'm looking for is the following:  cusID longestSeq 0 001 3 1 002 4  Although I wasn't able to write much code to solve this problem, here are some of my thoughts: First, it's obvious that I need to group the original DataFrame by cusID to analyse each of the two sequences separately. One idea I had was to apply some function to transform the DataFrame into this format:  cusID seq 0 001 [A, B, C, D, A, E, A] 1 002 [B, C, D, E]  and then working on each list separately, and use some sort of counter to take the maximum of the length of the paths that exclude A. My problem is transcribing that logic into code (if correct). Any help would be appreciated.

Answer: First step is to normalize the sequences. seqs = pd.concat([ df.drop(columns="end").rename(columns={"start":"node"}), df.groupby("cusID").tail(1).drop(columns="start").rename(columns={"end":"node"}) ]) seqs = seqs.sort_values("cusID", kind="mergesort").reset_index(drop=True) >>> seqs cusID node 0 001 A 1 001 B 2 001 C 3 001 D 4 001 A 5 001 E 6 001 A 7 002 B 8 002 C 9 002 D 10 002 E  Then, using zero_runs we define: def longest_non_a(seq): eqa = seq == "A" runs = zero_runs(eqa) return (runs[:,1] - runs[:,0]).max() result = seqs.groupby("cusID")["node"].apply(longest_non_a) >>> result cusID 001 3 002 4 Name: node, dtype: int64
Question: I want to make simple classifier with Keras that will classify my data. Features are numeric data and results are string/categorical data. I'm predicting 15 different categories/classes. This is how my code looks: model = Sequential() model.add(Dense(16, input_dim = x_train.shape[1], activation = 'relu')) # input layer requires input_dim param model.add(Dense(16, activation = 'relu')) model.add(Dense(16, activation = 'relu')) model.add(Dense(1, activation='relu')) model.compile(loss="binary_crossentropy", optimizer= "adam", metrics=['accuracy']) #es = EarlyStopping(monitor='loss', min_delta=0.005, patience=1, verbose=1, mode='auto') model.fit(x_train, y_train, epochs = 100, shuffle = True, batch_size=128, verbose=2) scores = model.evaluate(x_test, y_test) print(model.metrics_names[0], model.metrics_names[1])  the problem is that I'm always getting this error: ValueError: could not convert string to float 'category1'  What am I doing wrong? When I replace my classes names "category1", "category2" etc with integer numbers, my code works but it always give me accuracy of 0. I have tried to change number of nodes and layers and activation functions but the result is always 0. It seams like the model thinks that I'm doing regression not classification. What is the correct way to do classification with Keras lib If my categorical values are not just 1 or 0?

Answer: You need to convert your string categories to integers, there is a method for that: y_train = tf.keras.utils.to_categorical(y_train, num_classes=num_classes)  Also, the last layer for multi-class classification should be something like: model.add(Dense(NUM_CLASSES, activation='softmax'))  And finally, for multi-class classification, the correct loss would be categorial cross-entropy. model.compile(loss="categorical_crossentropy", optimizer= "adam", metrics=['accuracy'])  This is a nice example available from tensorflow: Classification Example
Question: How can a Jupyter notebook cell be split in VSCode? I.e., how to split a single cell with multiple lines into two cells with the top lines (above the cursor) in one cell and the bottom lines (below the cursor) in another cell? I've tried Cntrl Shift - using the Daily Insiders Python Extension, but it doesn't seem to do anything.

Answer: The Ctrl Shift - is for zooming out the display by default in VS Code. This feature has been put for a long time in Github, and the following is the request:  Jupyter Split Cell and Select Multiple Cells command  This issue is still open, although there's Notebooks are getting revamped! existed, it's for VS Code Insiders, not our current using VS Code. Maybe the production group can improve it in the future.
Question: In Julia, calling a function with the @edit macro from the REPL will open the editor and put the cursor at the line where the method is defined. So, doing this: julia> @edit 1 + 1  jumps to julia/base/int.jl and puts the cursor on the line: (+)(x::T, y::T) where {T<:BitInteger} = add_int(x, y)  As does the function form: edit(+, (Int, Int)) Is there an equivalent decorator/function in Python that does the same from the Python REPL?

Answer: Disclaimer: In the Python ecosystem, this is not the job of the core language/runtime but rather tools such as IDEs. For example, the ipython shell has the ?? special syntax to get improved help including source code. Python 3.8.5 (default, Jul 21 2020, 10:42:08) Type 'copyright', 'credits' or 'license' for more information IPython 7.18.1 -- An enhanced Interactive Python. Type '?' for help. In [1]: import random In [2]: random.uniform?? Signature: random.uniform(a, b) Source: def uniform(self, a, b): "Get a random number in the range [a, b) or [a, b] depending on rounding." return a + (b-a) * self.random() File: /usr/local/Cellar/python@3.8/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/random.py Type: method   The Python runtime itself allows viewing source code of objects via inspect.getsource. This uses a heuristic to search the source code as available; the objects themselves do not carry their source code. Python 3.8.5 (default, Jul 21 2020, 10:42:08) [Clang 11.0.0 (clang-1100.0.33.17)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import inspect >>> print(inspect.getsource(inspect.getsource)) def getsource(object): """Return the text of the source code for an object. The argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a single string. An OSError is raised if the source code cannot be retrieved.""" lines, lnum = getsourcelines(object) return ''.join(lines)  It is not possible to resolve arbitrary expressions or statements to their source; since all names in Python are resolved dynamically, the vast majority of expressions does not have a well-defined implementation unless executed. A debugger, e.g. as provided by pdb.set_trace(), allows inspecting the expression as it is executed.
Question: I've just gotten into writing discord bots. While trying to follow online instructions and tutorials, my bot would not respond to commands. It responded perfectly fine to on_message(), but no matter what I try it won't respond to commands. I'm sure it's something simple, but I would appreciate the help. import discord from discord.ext.commands import Bot from discord.ext import commands bot = commands.Bot(command_prefix='$') TOKEN = '<token-here>' @bot.event async def on_ready(): print(f'Bot connected as {bot.user}') @bot.event async def on_message(message): if message.content == 'test': await message.channel.send('Testing 1 2 3') @bot.command(name='go') async def dosomething(ctx): print("command called") #Tried putting this in help in debugging await message.channel.send("I did something") bot.run(TOKEN)  Picture of me prompting the bot and the results

Answer: Ok. First of all, the only import statement that you need at the top is from discord.ext import commands. The other two are not necessary. Second of all, I tried messing around with your code myself and found that the on_message() function seems to interfere with the commands so taking that out should help. Third of all, I only found this out when I duplicated one of my own working bots and slowly changed all of the code until it was identical to yours. For some reason python didn't like it when I just copied and pasted your code. I have never seen something like this before so I honestly don't know what to say other than that your code is correct and should work as long as you take the on_message() function out. Here's the final code that I got working: from discord.ext import commands bot = commands.Bot(command_prefix="$") TOKEN = "<token-here>" @bot.event async def on_ready(): print(f'Bot connected as {bot.user}') @bot.command() async def dosomething(ctx): await ctx.send("I did something") bot.run(TOKEN)  As you can see the only things that I have changed from your code are that I removed the redundant imports at the top, and I deleted the on_message() function. It works perfectly like this on my end so I would suggest that you re-type it out like this in a new file and see if that works. If that doesn't work for you then my next guess would be that there is a problem with your installation of discord.py so you might try uninstalling it and then reinstalling it. If none of that helps let me know and I will see if I can help you find anything else that might be the cause of the problem.
Question: In below code, I draw 2 3D graphs with plotly.graph_objects. I'm unable to put them together. import plotly.graph_objects as go import numpy as np pts = np.loadtxt(np.DataSource().open('https://raw.githubusercontent.com/plotly/datasets/master/mesh_dataset.txt')) x, y, z = pts.T ### First graph fig = go.Figure(data=[go.Mesh3d(x=x, y=y, z=z, alphahull=5, opacity=0.4, color='cyan')]) fig.show() ####### x = [0, 1, 0] y = [0, 2, 3] tvects = [x,y] orig = [0,0,0] df=[] coords = [[orig, np.sum([orig, v],axis=0)] for v in tvects] for i,c in enumerate(coords): X1, Y1, Z1 = zip(c[0]) X2, Y2, Z2 = zip(c[1]) vector = go.Scatter3d(x = [X1[0],X2[0]], y = [Y1[0],Y2[0]], z = [Z1[0],Z2[0]], marker = dict(size = [0,5], color = ['blue'], line=dict(width=5, color='DarkSlateGrey')), name = 'Vector'+str(i+1)) data.append(vector) ### Second graph fig = go.Figure(data=data) fig.show() ########  Could you please elaborate on how to do so, and how to have the arrow with head rather than a line in the second graph?

Answer: If you'd like to build on your first fig definition, just include the following after your first call to go.Figure() data = fig._data  data is now a list which will have new elements appended to it in the rest of your already exising code under: for i,c in enumerate(coords): X1, Y1, Z1 = zip(c[0]) X2, Y2, Z2 = zip(c[1]) vector = go.Scatter3d(<see details in snippet below>) data.append(vector)  Result:  Regarding the arrows, the only options dierectly available to you through go.Scatter3D are: ['circle', 'circle-open', 'square', 'square-open','diamond', 'diamond-open', 'cross', 'x']  I hope one of those options will suit your needs. You can specify which one in: marker = dict(size = [15,15], color = ['blue'], symbol = 'diamond', line=dict(width=500, #color='red' )),  Complete code: import plotly.graph_objects as go import numpy as np pts = np.loadtxt(np.DataSource().open('https://raw.githubusercontent.com/plotly/datasets/master/mesh_dataset.txt')) x, y, z = pts.T ### First graph fig = go.Figure(data=[go.Mesh3d(x=x, y=y, z=z, alphahull=5, opacity=0.4, color='cyan')]) #fig.show() ####### data = fig._data x = [0, 1, 0] y = [0, 2, 3] tvects = [x,y] orig = [0,0,0] df=[] coords = [[orig, np.sum([orig, v],axis=0)] for v in tvects] # ['circle', 'circle-open', 'square', 'square-open','diamond', 'diamond-open', 'cross', 'x'] for i,c in enumerate(coords): X1, Y1, Z1 = zip(c[0]) X2, Y2, Z2 = zip(c[1]) vector = go.Scatter3d(x = [X1[0],X2[0]], y = [Y1[0],Y2[0]], z = [Z1[0],Z2[0]], marker = dict(size = [15,15], color = ['blue'], symbol = 'diamond', line=dict(width=500, #color='red' )), name = 'Vector'+str(i+1)) data.append(vector) ### Second graph fig = go.Figure(data=data) fig.show() ########
Question: Suppose that I have a mask of an object and a point. I want to find the closest point of the object mask to the point. For example, in my drawing, there is an object, the blue shape in the image (assume inside is also the part of the object mask). And the red point is the point from which I want to find the closest distance to the object mask.  So, I want to find the thick green line as it is the shortest distance to the mask not the other ones (pink, orange, etc). I can do this using one of the following ways:  An inefficient way is to find the distances of all the pixels to the point using something like this (brute-force). Another way is to create many lines towards the mask with epsilon angle difference and find the closest point over that line, which is also not very nice. I can create lines over the edge, and find the closest point of each of those lines on the object border. (This may not be as easy as I thought, first I need to find the outer border, etc.)  But none of these methods are is elegant. I am wondering what is a more elegant and the most efficient way to determine this?

Answer: You could do a sort of binary search:  let's call P your point and consider circles centred on P pick any point M on the mask, the circle through M will intersect the mask now repeat until convergence, if circle intersects mask, reduce radius otherwise increase it (by binary search type amounts)  This will not work if your mask is not nicely connected, but then if that's not the case I doubt you can do better than brute force... The total cost should be linear for the circle-mask intersection check times log for the binary search.
Question: I have a np.arrray like this: [[ 1.3 , 2.7 , 0.5 , NaN , NaN], [ 2.0 , 8.9 , 2.5 , 5.6 , 3.5], [ 0.6 , 3.4 , 9.5 , 7.4 , NaN]]  And a function to compute the distance between two rows: def nan_manhattan(X, Y): nan_diff = np.absolute(X - Y) length = nan_diff.size return np.nansum(nan_diff) * length / (length - np.isnan(nan_diff).sum())  I need all pairwise distances, and I don't want to use a loop. How do I do that?

Answer: Leveraging broadcasting - def manhattan_nan(a): s = np.nansum(np.abs(a[:,None,:] - a), axis=-1) m = ~np.isnan(a) k = m.sum(1) r = a.shape[1]/np.minimum.outer(k,k) out = s*r return out  Benchmarking From OP's comments, the use-case seems to be a tall array. Let's reproduce one for benchmarking re-using given sample data : In [2]: a Out[2]: array([[1.3, 2.7, 0.5, nan, nan], [2. , 8.9, 2.5, 5.6, 3.5], [0.6, 3.4, 9.5, 7.4, nan]]) In [3]: a = np.repeat(a, 100, axis=0) # @Dani Mesejo's soln In [4]: %timeit pdist(a, nan_manhattan) 1.02 s ± 35.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # Naive for-loop version In [18]: n = a.shape[0] In [19]: %timeit [[nan_manhattan(a[i], a[j]) for i in range(j+1,n)] for j in range(n)] 991 ms ± 45.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) # With broadcasting In [9]: %timeit manhattan_nan(a) 8.43 ms ± 49.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)