eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I want to save X (ndarray) with dimensions (3960, 225) in excel file (.xlsx). In X I have some missing values (nan). I made a code for it. However, I am getting the error. Here is the Code: workbook = xlsxwriter.Workbook('arrays.xlsx') worksheet = workbook.add_worksheet() row = 0 for col, data in enumerate(X): worksheet.write_column(row, col, data) workbook.close() df = pd.DataFrame(X) ## save to xlsx file filepath = 'my_excel_file.xlsx' df.to_excel(filepath, index=False)  Here is the traceback: Traceback (most recent call last): File "C:\Program Files\JetBrains\PyCharm 2020.2.1\plugins\python\helpers\pydev\pydevd.py", line 1448, in _exec pydev_imports.execfile(file, globals, locals) # execute the script File "C:\Program Files\JetBrains\PyCharm 2020.2.1\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "C:/Users/Nafees Ahmed/PycharmProjects/Extra_Sensory_Experimetns/main.py", line 475, in <module> worksheet.write_column(row, col, data) File "C:\Users\Nafees Ahmed\AppData\Local\Programs\Python\Python38\lib\site-packages\xlsxwriter\worksheet.py", line 69, in cell_wrapper return method(self, *args, **kwargs) File "C:\Users\Nafees Ahmed\AppData\Local\Programs\Python\Python38\lib\site-packages\xlsxwriter\worksheet.py", line 1164, in write_column error = self._write(row, col, token, cell_format) File "C:\Users\Nafees Ahmed\AppData\Local\Programs\Python\Python38\lib\site-packages\xlsxwriter\worksheet.py", line 481, in _write return self._write_number(row, col, *args) File "C:\Users\Nafees Ahmed\AppData\Local\Programs\Python\Python38\lib\site-packages\xlsxwriter\worksheet.py", line 589, in _write_number raise TypeError( TypeError: NAN/INF not supported in write_number() without 'nan_inf_to_errors' Workbook() option  Perhaps, it is coming due to nan (missing) values. Is there any simple way to handle this error?

Answer: Filling NaN values with zero, does not solve the problem, If you want to keep NaN values as NaN, you should skip filling value in like that: row = 0 for col, data in enumerate(X): try: worksheet.write_column(row, col, data) except: pass
Question: I have a data matrix a and I have list of indices stored in array idx. I would like to get 10-length data starting at each of the indices defined by idx . Right now I use a for loop to achieve this. But it is extremely slow as I have to do this data fetch about 1000 times in an iteration. Below is a minimum working example. import numpy as np a = np.random.random(1000) idx = np.array([1, 5, 89, 54]) # I want "data" array to have np.array([a[1:11], a[5:15], a[89:99], a[54:64]]) # I use for loop below but it is slow data = [] for id in idx: data.append(a[id:id+10]) data = np.array(data)  Is there anyway to speed up this process? Thanks. EDIT: My question is different from the question asked here. In the question, the size of the chunks is random in contrast to fixed chunk size in my question. Other differences exist. I do not have to use up the entire array a and an element can occur in more than one chunk. My question does not necessarily "split" the array.

Answer: (Thanks to suggestion from @MadPhysicist) This should work: a[idx.reshape(-1, 1) + np.arange(10)]  Output: Shape (L,10), where L is the length of idx Notes:  This does not check for index-out-of-bound situations. I suppose it's easy to first ensure that idx doesn't contain such values.  Using np.take(a, idx.reshape(-1, 1) + np.arange(10), mode='wrap') is an alternative, that will handle out-of-bounds indices by wrapping them around a. Passing mode='clip' instead of mode='wrap' would clip the excessive indices to the last index of a. But then, np.take() would probably have a completely different perf. characteristic / scaling characteristic.
Question: I'm trying to embed an interactive plotly (or bokeh) plot into excel. To do this I've tried the following three things:  embed a Microsoft Web Browser UserForm into excel, following:   How do I embed a browser in an Excel VBA form?  This works and enables both online and offline html to be loaded  creating a plotly html  ''' import plotly import plotly.graph_objects as go x = [0.1, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0] y = [i**2 for i in x] fig = go.Figure() fig.add_trace(go.Scatter(x=x, y=x, mode='markers', name="y=x", marker=dict(color='royalblue', size=8))) fig.add_trace(go.Scatter(x=x, y=y, name="y=x^2", line=dict(width=3))) plotly.offline.plot(fig, filename='C:/Users/.../pythonProject/test1.html')   repointing the webbrowser object in excel using .Navigate to the local plotly.html. Banner pops up with  ".... restricted this file from showing active content that could access your computer" clicking on the banner, I run into this error:  The same HTML can be opened in a browser. Is there any way to show interactive plots in excel?

Answer: Finally, I have managed to bring the interactive plot to excel after a discussion from Microsoft QnA and Web Browser Control & Specifying the IE Version To insert a Microsoft webpage to excel you have to change the compatibility Flag in the registry editor Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office\ClickToRun\REGISTRY\MACHINE\Software\Wow6432Node\Microsoft\Office\16.0\Common\COM Compatibility{8856F961-340A-11D0-A96B-00C04FD705A2} Change the DWord 0 instead of 400 Now you can insert the web browser object to excel, Step by step details are here Edit the HTML File generated from plotly manually by adding a tag for Using the X-UA-Compatible HTML Meta Tag Originally generated HTML file from plotly looks like this <html> <head><meta charset="utf-8" /></head> <body>  Modified HTML with browser compatibility <html> <head> <meta charset="utf-8" /> <meta http-equiv="X-UA-Compatible" content="IE=edge" /> </head> <body>  After this, I can able to view the interactive plot in excel also able to do the interactions same as a web browser  Macro used: Sub Button4_Click() ActiveSheet.WebBrowser1.Navigate "file:///C:/Users/vignesh.rajendran/Desktop/test5.html" End Sub
Question: Is it possible to display an HTML file at the endpoint? For example the home page then the user is visiting "/"?

Answer: Yes, it's possible FastAPI has HTMLResponse. You can return a HTMLResponse from fastapi import FastAPI from fastapi.responses import HTMLResponse app = FastAPI() @app.get("/", response_class=HTMLResponse) async def read_items(): html_content = """ <html> <head> <title>Some HTML in here</title> </head> <body> <h1>Look ma! HTML!</h1> </body> </html> """ return HTMLResponse(content=html_content, status_code=200)  You can also render templates with Jinja2 from fastapi import FastAPI, Request from fastapi.responses import HTMLResponse from fastapi.staticfiles import StaticFiles from fastapi.templating import Jinja2Templates app = FastAPI() app.mount("/static", StaticFiles(directory="static"), name="static") templates = Jinja2Templates(directory="templates") @app.get("/items/{id}", response_class=HTMLResponse) async def read_item(request: Request, id: str): return templates.TemplateResponse("item.html", {"request": request, "id": id}
Question: I have tried to staring the application in VSCODE by Python3. This is the code: from tkinter import * window = Tk() window.mainloop()  only 3 lines :)), but when I'm trying to execute the file in terminal it will give me an error, which you can see below. arash@Arash-MacBook-Pro tkinter % python3 main.py macOS 11 or later required ! zsh: abort ------ python3 main.py arash@Arash-MacBook-Pro tkinter %  My Mac worked on "Big Sur" and has Python3.9.

Answer: This is an issue in the way brew installs Python (source). If you install Python directly via the official installer here then tkinter should work as expected.
Question: I just installed tensorflow v2.3 on anaconda python. I tried to test out the installation using the python command below; $ python -c "import tensorflow as tf; x = [[2.]]; print('tensorflow version', tf.__version__); print('hello, {}'.format(tf.matmul(x, x)))"  I got the following message; 2020-12-15 07:59:12.411952: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. hello, [[4.]]  From the message, it seems that the installation was installed successfully. But what does This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX AVX2 mean exactly? Am I using a tensorflow version with some limited features? Any side effects? I am using Windows 10.

Answer: An important part of Tensorflow is that it is supposed to be fast. With a suitable installation, it works with CPUs, GPUs, or TPUs. Part of going fast means that it uses different code depending on your hardware. Some CPUs support operations that other CPUs do not, such as vectorized addition (adding multiple variables at once). Tensorflow is simply telling you that the version you have installed can use the AVX and AVX2 operations and is set to do so by default in certain situations (say inside a forward or back-prop matrix multiply), which can speed things up. This is not an error, it is just telling you that it can and will take advantage of your CPU to get that extra speed out. Note: AVX stands for Advanced Vector Extensions.
Question: I want to get the latest Python source from https://www.python.org/ftp/python/. While posting this, the latest version is 3.9.1. I do not want to hardcode 3.9.1 in my code to get the latest version and keep on updating the version when a new version comes out. I am using Ubuntu 16.04. Is there a programmatic way to get the latest Python version (using curl) and use that version to get the latest source?

Answer: I had a similar problem and couldn't find anything better than scraping the downloads page. You mentioned curl, so I'm assuming you want a shell script. I ended up with this: url='https://www.python.org/ftp/python/' curl --silent "$url" | sed -n 's!.*href="\([0-9]\+\.[0-9]\+\.[0-9]\+\)/".*!\1!p' | sort -rV | while read -r version; do filename="Python-$version.tar.xz" # Versions which only have alpha, beta, or rc releases will fail here. # Stop when we find one with a final release. if curl --fail --silent -O "$url/$version/$filename"; then echo "$filename" break fi done  This relies on sort -V, which I believe is specific to GNU coreutils. That shouldn't be a problem on Ubuntu. If you're using this in a larger script and want to use the version or filename variables after the loop, see How to pipe input to a Bash while loop and preserve variables after loop ends.
Question: I'm trying to create a bar chart using plotly in python, which is both stacked and grouped. Toy example (money spent and earned in different years): import pandas as pd import plotly.graph_objs as go data = pd.DataFrame( dict( year=[2000,2010,2020], var1=[10,20,15], var2=[12,8,18], var3=[10,17,13], var4=[12,11,20], ) ) fig = go.Figure( data = [ go.Bar(x=data['year'], y=data['var1'], offsetgroup=0, name='spent on fruit'), go.Bar(x=data['year'], y=data['var2'], offsetgroup=0, base=data['var1'], name='spent on toys'), go.Bar(x=data['year'], y=data['var3'], offsetgroup=1, name='earned from stocks'), go.Bar(x=data['year'], y=data['var4'], offsetgroup=1, base=data['var3'], name='earned from gambling'), ] ) fig.show()  The result seems fine at first:  But watch what happens when I turn off e.g. "spent on fruit":  The "spent on toys" trace remains floating instead of starting from 0. Can this be fixed? or maybe the whole offsetgroup + base approach won't work here. But what else can I do? Thanks! Update: according to this Github issue, stacked, grouped bar plots are being developed for future plotly versions, so this probably won't be an issue anymore.

Answer: There doesn't seem to be a way to create both stacked and grouped bar charts in Plotly, but there is a workaround that might resolve your issue. You will need to create subgroups, then use a stacked bar in Plotly to plot the bars one at a time, plotting var1 and var2 with subgroup1, and var3 and var4 with subgroup2. This solution gives you the functionality you want, but changes the formatting and aesthetic of the bar chart. There will be equal spacing between each bar as from Plotly's point of view these are stacked bars (and not grouped bars), and I couldn't figure out a way to eliminate the subgroup1 and subgroup2 text without also getting rid of the years in the x-axis ticks. Any Plotly experts please feel free to chime in and improve my answer! import pandas as pd import plotly.graph_objs as go df = pd.DataFrame( dict( year=[2000,2010,2020], var1=[10,20,15], var2=[12,8,18], var3=[10,17,13], var4=[12,11,20], ) ) fig = go.Figure() fig.update_layout( template="simple_white", xaxis=dict(title_text="Year"), yaxis=dict(title_text="Count"), barmode="stack", ) groups = ['var1','var2','var3','var4'] colors = ["blue","red","green","purple"] names = ['spent on fruit','spent on toys','earned from stocks','earned from gambling'] i = 0 for r, n, c in zip(groups, names, colors): ## put var1 and var2 together on the first subgrouped bar if i <= 1: fig.add_trace( go.Bar(x=[df.year, ['subgroup1']*len(df.year)], y=df[r], name=n, marker_color=c), ) ## put var3 and var4 together on the first subgrouped bar else: fig.add_trace( go.Bar(x=[df.year, ['subgroup2']*len(df.year)], y=df[r], name=n, marker_color=c), ) i+=1 fig.show()
Question: From PyTorch documentation: b = torch.rand(10, requires_grad=True).cuda() b.is_leaf False # b was created by the operation that cast a cpu Tensor into a cuda Tensor e = torch.rand(10).cuda().requires_grad_() e.is_leaf True # e requires gradients and has no operations creating it f = torch.rand(10, requires_grad=True, device="cuda") f.is_leaf True # f requires grad, has no operation creating it  But why are e and f leaf Tensors, when they both were also cast from a CPU Tensor, into a Cuda Tensor (an operation)? Is it because Tensor e was cast into Cuda before the in-place operation requires_grad_()? And because f was cast by assignment device="cuda" rather than by method .cuda()?

Answer: When a tensor is first created, it becomes a leaf node. Basically, all inputs and weights of a neural network are leaf nodes of the computational graph. When any operation is performed on a tensor, it is not a leaf node anymore. b = torch.rand(10, requires_grad=True) # create a leaf node b.is_leaf # True b = b.cuda() # perform a casting operation b.is_leaf # False  requires_grad_() is not an operation in the same way as cuda() or others are. It creates a new tensor, because tensor which requires gradient (trainable weight) cannot depend on anything else. e = torch.rand(10) # create a leaf node e.is_leaf # True e = e.cuda() # perform a casting operation e.is_leaf # False e = e.requires_grad_() # this creates a NEW tensor e.is_leaf # True  Also, detach() operation creates a new tensor which does not require gradient: b = torch.rand(10, requires_grad=True) b.is_leaf # True b = b.detach() b.is_leaf # True  In the last example we create a new tensor which is already on a cuda device. We do not need any operation to cast it. f = torch.rand(10, requires_grad=True, device="cuda") # create a leaf node on cuda
Question: i'm trying to find theses two horizontal lines with the Houghlines transform. As you can see, the picture is very noisy ! Currently my workflow looks like this :  crop the image  blur it  low the noise (for that, I invert the image, and then substract the blured image to the inverted one)  open it and dilate it with an "horizontal kernel" (kernel_1 = np.ones((10,1), np.uint8)  threshold  Houglines   the results are not as good as expected... is there a better strategy, knowing that I will always serach for horizontal lines (hence, abs(theta) will always be closed to 0 or pi)  

Answer: the issue is the noise and the faint signal. you can subdue the noise with averaging/integration, while maintaining the signal because it's replicated along a dimension (signal is a line). your approach using a very wide but narrow kernel can be extended to simply integrating along the whole image.  rotate the image so the suspected line is aligned with an axis (let's say horizontal) sum up all pixels of one scanline (horizontal line), np.sum(axis=1) or mean, either way mind the data type. working with floats is convenient. work with the 1-dimensional series of values.  this will not tell you how long the line is, only that it's there and potentially spanning the whole width. edit: since my answer got a reaction, I'll elaborate as well: I think you can lowpass that to get the "gray" baseline, then subtract ("difference of gaussians"). that should give you a nice signal. import numpy as np import cv2 as cv import matplotlib.pyplot as plt import scipy.ndimage im = cv.imread("0gczo.png", cv.IMREAD_GRAYSCALE) / np.float32(255) relief = im.mean(axis=1) smoothed = scipy.ndimage.gaussian_filter(relief, sigma=2.0) baseline = scipy.ndimage.gaussian_filter(relief, sigma=10.0) difference = smoothed - baseline std = np.std(difference) level = 2 outliers = (difference <= std * -level) plt.plot(difference) plt.hlines([std * +level, std * -level], xmin=0, xmax=len(relief)) plt.plot(std * -level + outliers * std) plt.show() # where those peaks are: edgemap = np.diff(outliers.astype(np.int8)) (edges,) = edgemap.nonzero() print(edges) # [392 398 421 427] print(edgemap[edges]) # [ 1 -1 1 -1]
Question: My problem is kind of simple, but I'm not sure there's a way to do what I'm looking for: I had to store in a SQL database some data, that includes some intervals that will later be used. Because of this, I had to store it as a string, like this:  variable interval A (-0.001, 2.0] A (2.0, 6.0]  So, then, I want to use said intervals to cut another variable, like this: df1 = pd.DataFrame({'interval': {4: '(-0.001, 2.0]', 5: '(2.0, 6.0]'}, 'variable': {4: 'A', 5: 'A', }}) df2 = pd.DataFrame({'A': [1,1,3]}) bins = df1[df1.variable.eq('A')].interval new_series = pd.cut(df2['A'], bins=bins)  But this brings:  ValueError: could not convert string to float: '(-0.001, 2.0]'  Tried: bins = bins.astype('interval')  But this brings: TypeError: type <class 'str'> with value (-0.001, 2.0] is not an interval  Is there something I can do? Thanks

Answer: IIUC, you could parse the string by hand, then convert bins to IntervalIndex: import ast import pandas as pd def interval_type(s): """Parse interval string to Interval""" table = str.maketrans({'[': '(', ']': ')'}) left_closed = s.startswith('[') right_closed = s.endswith(']') left, right = ast.literal_eval(s.translate(table)) t = 'neither' if left_closed and right_closed: t = 'both' elif left_closed: t = 'left' elif right_closed: t = 'right' return pd.Interval(left, right, closed=t) df1 = pd.DataFrame({'interval': {4: '(-0.001, 2.0]', 5: '(2.0, 6.0]'}, 'variable': {4: 'A', 5: 'A'}}) df1['interval'] = df1['interval'].apply(interval_type) df2 = pd.DataFrame({'A': [1, 1, 3]}) bins = df1[df1.variable.eq('A')].interval new_series = pd.cut(df2['A'], bins=pd.IntervalIndex(bins)) print(new_series)  Output 0 (-0.001, 2.0] 1 (-0.001, 2.0] 2 (2.0, 6.0] Name: A, dtype: category Categories (2, interval[float64]): [(-0.001, 2.0] < (2.0, 6.0]]
Question: I am displaying an image of a molecule using IPython.display in Jupyter.  The resolution of the image is quite low. Is there a way to specify the width and height of the displayed image and its resolution? I googled it and could not find anything. All I need is something like this: display(moleSmilemol, format='svg', width=1000, height=1000)  Any pointers would be appreciated. Update: I could add custom css, which will blow up the picture that was generate, but it is still low quality. I am interested increasing the quality of the picture and its size too. So something deeper than CSS is needed.

Answer: Try changing the variables in the rdkit.Chem.Draw.IPythonConsole module: from rdkit.Chem.Draw import IPythonConsole IPythonConsole.molSize = (800, 800) # Change image size IPythonConsole.ipython_useSVG = True # Change output to SVG mol = Chem.MolFromSmiles('N#Cc1cccc(-c2nc(-c3cccnc3)no2)c1') display(mol)  Otherwise, you need to use the rdMolDraw2D module to create the drawings yourself with the parameters you require.
Question: In R I can have a data.frame or a list with several arguments, and I can operate on them using the with function. For example: d <- data.frame(x = 1:3, y = 2:4, z = 3:5) # I can use: d$x+d$y*d$z-5 # Or, more simply, I can use: with(d, x+y*z-5) # [1] 2 9 18  In pandas DataFrame I can use: d = {'x': [1, 2, 3], 'y': [2, 3, 4], 'z': [3, 4, 5]} df = pd.DataFrame(data=d) df.x+df.y*df.z-5 # 0 2 # 1 9 # 2 18 # dtype: int64  But is there a way to do some "with" like statement?

Answer: One idea is use DataFrame.eval if need processing some columns names some simple arithmetic operations: print (df.x+df.y*df.z-5) 0 2 1 9 2 18 dtype: int64 print (df.eval('x+y*z-5')) 0 2 1 9 2 18 dtype: int64
Question: I've noticed that asyncio/init.py from python 3.6 uses the following construct: from .base_events import * ... __all__ = (base_events.__all__ + ...)  The base_events symbol is not imported anywhere in the source code, yet the module still contains a local variable for it. I've checked this behavior with the following code, put into an __init__.py with a dummy test.py next to it: test = "not a module" print(test) from .test import * print(test)   not a module  <module 'testpy.test' from 'C:\Users\MrM\Desktop\testpy\test.py'>  Which means that the test variable got shadowed after using a star import. I fiddled with it a bit, and it turns out that it doesn't have to be a star import, but it has to be inside an __init__.py, and it has to be relative. Otherwise the module object is not being assigned anywhere. Without the assignment, running the above example from a file that isn't an __init__.py will raise a NameError. Where is this behavior coming from? Has this been outlined in the spec for import system somewhere? What's the reason behind __init__.py having to be special in this way? It's not in the reference, or at least I couldn't find it.

Answer: This behavior is defined in The import system documentation section 5.4.2 Submodules  When a submodule is loaded using any mechanism (e.g. importlib APIs, the import or import-from statements, or built-in import()) a binding is placed in the parent module’s namespace to the submodule object. For example, if package spam has a submodule foo, after importing spam.foo, spam will have an attribute foo which is bound to the submodule.  A package namespace includes the namespace created in __init__.py plus extras added by the import system. The why is for namespace consistency.  Given Python’s familiar name binding rules this might seem surprising, but it’s actually a fundamental feature of the import system. The invariant holding is that if you have sys.modules['spam'] and sys.modules['spam.foo'] (as you would after the above import), the latter must appear as the foo attribute of the former.
Question: I want to use Window functions on sqlite3 on my python3.8 code running on AWS Lambda. They are available since version 3.25. Unfortunately, on AWS Lambda Python3.8, sqlite3 library is outdated: >>> sqlite3.sqlite_version '3.7.17'  while locally, on my homebrew install of Python3.8: (working) >>> import sqlite3 >>> sqlite3.sqlite_version '3.31.1'  How can I get an sqlite3 version > 3.25 on AWS Lambda Python 3.8 ?

Answer: I found a way: I used the external package pysqlite3, in the binary version. in my requirements.txt pysqlite3-binary==0.4.4  in the code try: import pysqlite3 as sqlite3 except ModuleNotFoundError: import sqlite3 # for local testing because pysqlite3-binary couldn't be installed on macos