eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm new with neural networks. I wanted to make a custom loss function in TensorFlow, but I need to get a vector of weights, so I did it in this way: def my_loss(weights): def custom_loss(y, y_pred): return weights*(y - y_pred) return custom_loss model.compile(optimizer='adam', loss=my_loss(weights), metrics=['accuracy']) model.fit(x_train, y_train, batch_size=None, validation_data=(x_test, y_test), epochs=100)  When I launch it, I receive this error: InvalidArgumentError: Incompatible shapes: [50000,10] vs. [32,10]  The shapes are: print(weights.shape) print(y_train.shape)  (50000, 10) (50000, 10)  So I thought that it was a problem with the batches, I don't have a strong background with TensorFlow, so I tried to solve in a naive way using a global variable batch_index = 0  and then updating it within a custom callback into the "on_batch_begin" hook. But it didn't work and it was a horrible solution. So, how can I get the exact part of the weights with the corresponding y? Do I have a way to get the current batch index inside the custom loss? Thank you in advance for your help

Answer: this is a workaround to pass additional arguments to a custom loss function, in your case an array of weights. the trick consists in using fake inputs which are useful to build and use the loss in the correct ways. don't forget that keras handles fixed batch dimension I provide a dummy example in a regression problem def mse(y_true, y_pred, weights): error = y_true-y_pred return K.mean(K.square(error) + K.sqrt(weights)) X = np.random.uniform(0,1, (1000,10)) y = np.random.uniform(0,1, 1000) w = np.random.uniform(0,1, 1000) inp = Input((10,)) true = Input((1,)) weights = Input((1,)) x = Dense(32, activation='relu')(inp) out = Dense(1)(x) m = Model([inp,true,weights], out) m.add_loss( mse( true, out, weights ) ) m.compile(loss=None, optimizer='adam') m.fit(x=[X, y, w], y=None, epochs=3) ## final fitted model to compute predictions (remove W if not needed) final_m = Model(inp, out)
Question: I was reading the scikitlearn tutorial about column transformer. The given example (https://scikit-learn.org/stable/modules/generated/sklearn.compose.make_column_selector.html#sklearn.compose.make_column_selector) works, but when I tried to select only few columns, It gives me error. MWE import numpy as np import pandas as pd import seaborn as sns from sklearn.compose import make_column_transformer from sklearn.compose import make_column_selector df = sns.load_dataset('tips') mycols = ['tip','sex'] ct = make_column_transformer(make_column_selector(pattern=mycols) ct.fit_transform(df)  Required I want only the select columns in the output. NOTE Of course, I know I can do df[mycols], I am looking for scikit learn pipeline example.

Answer: If you don't mind mlxtend, it has built-in transformer for that. Using mlxtend from mlxtend.feature_selection import ColumnSelector pipe = ColumnSelector(mycols) pipe.fit_transform(df)  For sklearn >= 0.20  Reference: https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html  from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline import seaborn as sns df = sns.load_dataset('tips') mycols = ['tip','sex'] pipeline = Pipeline([ ("selector", ColumnTransformer([ ("selector", "passthrough", mycols) ], remainder="drop")) ]) pipeline.fit_transform(df)  For sklearn < 0.20 from sklearn.base import BaseEstimator, TransformerMixin from sklearn.pipeline import Pipeline class FeatureSelector(BaseEstimator, TransformerMixin): def __init__(self, columns): self.columns = columns def fit(self, X, y=None): return self def transform(self, X, y=None): return X[self.columns] pipeline = Pipeline([('selector', FeatureSelector(columns=mycols)) ]) pipeline.fit_transform(df)[:5]
Question: I am writing a custom Django migration script. As per the django docs on custom migrations, I should be able to use my model vis-a-vis apps.get_model(). However, when trying to do this I get the following error: AttributeError: type object 'MyModel' has no attribute 'objects'  I think this has to do with the apps registry not being ready, but I am not sure. Sample code: def do_thing(apps, schema_editor): my_model = apps.get_model('app', 'MyModel') objects_ = my_model.objects.filter( some_field__isnull=True).prefetch_related( 'some_field__some_other_field') # exc raised here class Migration(migrations.Migration): atomic = False dependencies = [ ('app', '00xx_auto_xxx') ] operations = [ migrations.RunPython(do_thing), ]  A simple print statement of apps.get_model()'s return value shows the following: <class '__fake__.MyModel'>. I'm not sure what this is, and if it is a result of not being ready. EDIT: I couldn't find any resources to explain why I am getting a __fake__ object so I decided to tinker with the code. I got it to work by preempting apps from args, as can be seen here: def do_thing(apps, schema_editor): from django.apps import apps my_model = apps.get_model('app', 'MyModel') objects_ = my_model.objects.filter( some_field__isnull=True).prefetch_related( 'some_field__some_other_field') # no more exc raised here  I am still confused and any help would be appreciated.

Answer: The fake objects are historical models. Here's the explanation from Django docs:  When you run migrations, Django is working from historical versions of your models stored in the migration files. [...] Because it’s impossible to serialize arbitrary Python code, these historical models will not have any custom methods that you have defined. They will, however, have the same fields, relationships, managers (limited to those with use_in_migrations = True) and Meta options (also versioned, so they may be different from your current ones).  In case objects is a custom manager, you can set use_in_migrations = True to make it available in migrations.
Question: I have specific problem with pandas: I need to select rows in dataframe which start with specific letters. Details: I've imported my data to dataframe and selected columns that I need. I've also narrowed it down to row index I need. Now I also need to select rows in other column where objects START with letters 'pl'.   Is there any solution to select row only based on first two characters in it? I was thinking about  pl = df[‘Code’] == pl*  but it won't work due to row indexing. Advise appreciated! 

Answer: If you use a string method on the Series that should return you a true/false result. You can then use that as a filter combined with .loc to create your data subset. new_df = df.loc[df[‘Code’].str.startswith('pl')].copy()
Question: I'm trying to bind the SHIFT+TAB keys, but I can't seem to get it to work. The widget I'm binding to is an Entry widget. I've tried binding the keys with widget.bind('<Shift_Tab>', func), but I get an error message saying:  File "/usr/lib64/python3.8/tkinter/init.py", line 1337, in _bind self.tk.call(what + (sequence, cmd)) _tkinter.TclError: bad event type or keysym "Shift_Tab"   Update I'm still having a problem detecting SHIFT+TAB. Here is my test code. My OS is Linux. The tab key works, just not SHIFT+TAB. Seems like a simple problem to solve, so I must be going about it wrong?  I'm trying to tab between columns in a Treeview that I have overlaid widgets on a row to simulate inline editing. There can be only one active widget on a line. I keep track of what column I'm in and when the user presses SHIFT+TAB or TAB, I remove the current widget and display a new widget in the corresponding column.  Here is a link to the complete project:  The project is in one file and has no imports. The code below is my attempt and it doesn't work. import tkinter as tk import tkinter.ttk as ttk class App(tk.Tk): def __init__(self): super().__init__() self.rowconfigure(0, weight=1) self.columnconfigure(0, weight=1) self.title('Default Demo') self.geometry('420x200') wdg = ttk.Entry(self) wdg.grid() def tab(_): print('Tab pressed.') def shift_tab(_): print('Shift tab pressed.') wdg.bind('<Tab>', tab) wdg.bind('<Control-ISO_Left_Tab>', shift_tab) def main(): app = App() app.mainloop() if __name__ == '__main__': main() 

Answer: Not a direct answer and too long for a comment. You can solve your question by yourself with a simple trick, bind <Key> to a function, and print the key event argument passed to the bind function where you can see which key is pressed or not. Try multiple combinations of keys to see what is the state and what is their keysym or keycode. import tkinter as tk def key_press(evt): print(evt) root = tk.Tk() root.bind("<Key>", key_press) root.mainloop()  It will output the following for pressing SHIFT + TAB combo on macOS. <KeyPress event state=Shift keysym=Tab keycode=3145753 char='\x19' x=-5 y=-50>  Where,  state=Shift means a key event's state is on SHIFT.  keysym=Tab means tab key is pressed. If we just press SHIFT, the keysym will be Shift_L or Shift_R (shows Shift_L for both shift keys on mac).  keycode is an unique code for each key even for different key combos for example keycode for Left Shift is 131330 and keycode for TAB is 3145737 but when SHIFT + TAB is pressed the code is not the same to either, it is 3145753. (I'm not sure if these are the same code for every os but one can figure it out by printing them to the console)  Also, see all the event attributes.   Though bind '<Shift-Tab>' key combination works well on Mac, it can also be used like so... def key_press(evt): if evt.keycode==3145753: print('Shift+Tab is pressed')
Question: I am running this code from the tutorial here: https://keras.io/examples/vision/image_classification_from_scratch/ with a custom dataset, that is divided in 2 datasets as in the tutorial. However, I got this error: TypeError: Input 'filename' of 'ReadFile' Op has type float32 that does not match expected type of string.  I made this casting. I tried this:  is_jfif = str(tf.compat.as_bytes("JFIF")) in fobj.peek(10)  but nothing changed as far as the error I am trying all day to figure out how to solve it, without any success. Can someone help me? Thank you...

Answer: Simplest way I found is to create a subfolder and copy the files to that subfolder. i.e. Lets assume your files are 0.jpg, 1.jpg,2.jpg....2000.jpg and in directory named "patterns". Seems like the Keras API does not accept it as the files are named by numbers and for Keras it is in float32. To overcome this issue, either you can rename the files as one answer suggests, or you can simply create a subfolder under "patterns" (i.e. "patterndir"). So now your image files are under ...\patterns\patterndir Keras (internally) possibly using the subdirectory name and may be attaching it in front of the image file thus making it a string (sth like patterndir_01.jpg, patterndir_02.jpg) [Note this is my interpretation, does not mean that it is true] When you compile it this time, you will see that it works and you will get a compiler message as: Found 2001 files belonging to 1 classes. Using 1601 files for training. Found 2001 files belonging to 1 classes. Using 400 files for validation.  My code looks like this import tensorflow as tf from tensorflow import keras from tensorflow.keras import layers #Generate a dataset image_size = (28, 28) batch_size = 32 train_ds = tf.keras.preprocessing.image_dataset_from_directory( "patterns", validation_split=0.2, subset="training", seed=1337, image_size=image_size, batch_size=batch_size, ) val_ds = tf.keras.preprocessing.image_dataset_from_directory( "patterns", validation_split=0.2, subset="validation", seed=1337, image_size=image_size, batch_size=batch_size, )
Question: I am trying to find out the python version I am using in Databricks. To find out I tried  import sys print(sys.version)  And I got the output as 3.7.3 However when I went to Cluster --> SparkUI --> Environment I see that the cluster Python version is 2. Which version does this refer to ? When I tried running %sh python --version  I still get Python 3.7.3 Can there be a different python version for each worker / driver node ? Note: I am using a setup where there is 1 worker node and 1 driver node (2 nodes in total with the same spec) and Databricks Runtime Version is 6.5 ML

Answer: Update: This issue has been fixed. For new cluster: If you create a new cluster it will have python environment variable as 3. For existing clusters: You need to add in Environment Variables tab in Cluster Configuration > Advanced, it changes in the Environmental variable.  PYSPARK_PYTHON=/databricks/python3/bin/python3    Thanks for bringing this to our attention. This is a product-bug, currently I'm working with the product team to fix the issue asap.  The default Python version for clusters created using the UI is Python 3.  As part of repro, I had created Databricks Runtime Version: 6.5 ML and observed the same behaviour. Cluster --> SparkUI --> Environment shows incorrect version.
Question: I have a list of values like this, lst = [1, 2, 3, 4, 5, 6, 7, 8]  Desired Output: window size = 3 1 # first element in the list forward = [2, 3, 4] backward = [] 2 # second element in the list forward = [3, 4, 5] backward = [1] 3 # third element in the list forward = [4, 5, 6] backward = [1, 2] 4 # fourth element in the list forward = [5, 6, 7] backward = [1, 2, 3] 5 # fifth element in the list forward = [6, 7, 8] backward = [2, 3, 4] 6 # sixth element in the list forward = [7, 8] backward = [3, 4, 5] 7 # seventh element in the list forward = [8] backward = [4, 5, 6] 8 # eight element in the list forward = [] backward = [5, 6, 7]  Lets assume a window size of 4, now my desired output:  for each_element in the list, I want 4 values in-front and 4 values backward ignoring the current value.  I was able to use this to get sliding window of values but this also not giving me the correct required output. import more_itertools list(more_itertools.windowed([1, 2, 3, 4, 5, 6, 7, 8], n=3)) 

Answer: Code: arr = [1, 2, 3, 4, 5, 6, 7, 8] window = 3 for backward, current in enumerate(range(len(arr)), start = 0-window): if backward < 0: backward = 0 print(arr[current+1:current+1+window], arr[backward:current])  Output: [2, 3, 4], [] [3, 4, 5], [1] [4, 5, 6], [1, 2] [5, 6, 7], [1, 2, 3] [6, 7, 8], [2, 3, 4] [7, 8], [3, 4, 5] [8], [4, 5, 6] [], [5, 6, 7]  One Liner: print(dict([(e, (lst[i+1:i+4], lst[max(i-3,0):i])) for i,e in enumerate(last)]))  Output: {1: ([2, 3, 4], []), 2: ([3, 4, 5], [1]), 3: ([4, 5, 6], [1, 2]), 4: ([5, 6, 7], [1, 2, 3]), 5: ([6, 7, 8], [2, 3, 4]), 6: ([7, 8], [3, 4, 5]), 7: ([8], [4, 5, 6]), 8: ([], [5, 6, 7])}  Credit: thanks to suggestions from @FeRD and @Androbin, the solution now looks better
Question: I am trying to automate the downloading of files from a Google Drive shared folder. The contents of the folder change daily. The folder is shared to anyone with a link to the folder. My problem is that the query does not return the new files that I have not opened yet unless I open the new files in Google Drive. folder_id = 'xxxx...xxx' results = drive_service.files().list(q=f"parents in '{folder_id}' and trashed = false", fields="nextPageToken, files(id, name)").execute()  Is there a way to list files in a shared folder that I have not opened yet? I tried to negate the sharedWithMe field but it does not work.

Answer: Yes, there is, but you need to specify that you want to include shared folders into the search This you can do by setting includeItemsFromAllDrives and supportsAllDrives to true In python you can implement it with folder_id = 'xxxx...xxx' results = drive_service.files().list(supportsAllDrives=True, includeItemsFromAllDrives=True, q="parents in '{folder_id}' and trashed = false", fields = "nextPageToken, files(id, name)").execute()  UPDATE Mind that shared with anyone with a link means that until you open a file, it is not explicitly shared with you and thus won't show up as part of your file list. If you want to change this, you need to ask the owner to specify you as a viewer of the folder.
Question: Using Blender created this model   that can be seen in A-frame in this link  This model is great and it gives an overview of what I'm trying to accomplish here. Basically, instead of having the names, I'd have dots that symbolize one specific platform. The best way to achieve it with current state of the art, at my sight, is through Plotly 3D Scatter Plots. I've got the following scatterplot   import plotly.express as px import pandas as pd df = pd.read_csv('https://raw.githubusercontent.com/tiago-peres/immersion/master/Platforms_dataset.csv') fig = px.scatter_3d(df, x='Functionality ', y='Accessibility', z='Immersion', color='Platforms') fig.show()  that by going to this link you'll be able to click a button and open it in Colab  This nearly looks like the model. Yet, still in need to add three planes to the plot in specific locations. More precisely, in x=?, y=? and z=? (I'm using question mark because the value can be anything stablished). In other words, want to add three planes to that scatterplot  x = 10 y = 30 z = 40  In the documentation, what closely resembles what I want was 3D Surface Plots. I've done research and found two similar questions with R  Insert 2D plane into a 3D Plotly scatter plot in R Add Regression Plane to 3d Scatter Plot in Plotly 

Answer: I think you might be looking for the add_trace function in plotly so you can just create the surfaces and then add them to the figure: Also, note, there's definitely ways to simplify this code, but for a general idea: import plotly.express as px import pandas as pd import plotly.graph_objects as go import numpy as np fig = px.scatter_3d(df, x='Functionality ', y='Accessibility', z='Immersion', color='Platforms') bright_blue = [[0, '#7DF9FF'], [1, '#7DF9FF']] bright_pink = [[0, '#FF007F'], [1, '#FF007F']] light_yellow = [[0, '#FFDB58'], [1, '#FFDB58']] # need to add starting point of 0 to each dimension so the plane extends all the way out zero_pt = pd.Series([0]) z = zero_pt.append(df['Immersion'], ignore_index = True).reset_index(drop = True) y = zero_pt.append(df['Accessibility'], ignore_index = True).reset_index(drop = True) x = zero_pt.append(df['Functionality '], ignore_index = True).reset_index(drop = True) length_data = len(z) z_plane_pos = 40*np.ones((length_data,length_data)) fig.add_trace(go.Surface(x=x, y=y, z=z_plane_pos, colorscale=light_yellow, showscale=False)) fig.add_trace(go.Surface(x=x.apply(lambda x: 10), y=y, z = np.array([z]*length_data), colorscale= bright_blue, showscale=False)) fig.add_trace(go.Surface(x=x, y= y.apply(lambda x: 30), z = np.array([z]*length_data).transpose(), colorscale=bright_pink, showscale=False))
Question: RuntimeError: Given groups=1, weight of size [32, 3, 16, 16, 16], expected input[100, 16, 16, 16, 3] to have 3 channels, but got 16 channels instead This is the portion of code I think where the problem is.  def __init__(self): super(Lightning_CNNModel, self).__init__() self.conv_layer1 = self._conv_layer_set(3, 32) self.conv_layer2 = self._conv_layer_set(32, 64) self.fc1 = nn.Linear(2**3*64, 128) self.fc2 = nn.Linear(128, 10) # num_classes = 10 self.relu = nn.LeakyReLU() self.batch=nn.BatchNorm1d(128) self.drop=nn.Dropout(p=0.15) def _conv_layer_set(self, in_c, out_c): conv_layer = nn.Sequential( nn.Conv3d(in_c, out_c, kernel_size=(3, 3, 3), padding=0), nn.LeakyReLU(), nn.MaxPool3d((2, 2, 2)), ) return conv_layer def forward(self, x): out = self.conv_layer1(x) out = self.conv_layer2(out) out = out.view(out.size(0), -1) out = self.fc1(out) out = self.relu(out) out = self.batch(out) out = self.drop(out) out = self.fc2(out) return out  This is the code I am working on 

Answer: nn.Conv3d expects the input to have size [batch_size, channels, depth, height, width]. The first convolution expects 3 channels, but with your input having size [100, 16, 16, 16, 3], that would be 16 channels. Assuming that your data is given as [batch_size, depth, height, width, channels], you need to swap the dimensions around, which can be done with torch.Tensor.permute: # From: [batch_size, depth, height, width, channels] # To: [batch_size, channels, depth, height, width] input = input.permute(0, 4, 1, 2, 3)
Question: In pytorch, I want to write a tensor to a file and visually read the file contents. For example, consider T = torch.tensor([3,4,5,6]). I want to write the tensor T to a file, say file_T.txt, and want to visually read the contents of the file_T.txt, which will be 3,4,5 and 6. How can I achieve this?

Answer: You can use numpy: import numpy as np np.savetxt('my_file.txt', torch.Tensor([3,4,5,6]).numpy())
Question: The following doesn't execute foo and gives RuntimeWarning: coroutine 'foo' was never awaited # urls.py async def foo(data): # process data ... @api_view(['POST']) def endpoint(request): data = request.data.get('data') # How to call foo here? foo(data) return Response({}) 

Answer: Found a way to do it. Create another file bar.py in the same directory as urls.py. # bar.py def foo(data): // process data  # urls.py from multiprocessing import Process from .bar import foo @api_view(['POST']) def endpoint(request): data = request.data.get('data') p = Process(target=foo, args=(data,)) p.start() return Response({})
Question: Now I got some problem that I can't explain and fix. This is my first python module  TimeHelper.py from datetime import datetime def fun1(currentTime = datetime.now()): print(currentTime)  and another is  Main.py from TimeHelper import fun1 import time fun1() time.sleep(5) fun1()  When I run the Main.py, the out put is 2020-06-16 09:17:52.316714 2020-06-16 09:17:52.316714 My problem is why the time will be same in the result ? Is there any restrict when passing datetime.now() in to default parameter ?

Answer: I think I find the answer. Thanks for @user2864740 So I change my TimeHelper.py to this from datetime import datetime def fun1(currentTime = None): if currentTime is None: currentTime = datetime.now() print(currentTime)  and anything work in my expectation.
Question: I am unable to perform a standard in operation with a pre-defined list of items. I am looking to do something like this: # Construct a simple example frame from datatable import * df = Frame(V1=['A','B','C','D'], V2=[1,2,3,4]) # Filter frame to a list of items (THIS DOES NOT WORK) items = ['A','B'] df[f.V1 in items,:]  This example results in the error:  TypeError: A boolean value cannot be used as a row selector Unfortunately, there doesn't appear to be a built-in object for in operations. I would like to use something like the %in% operator that is native to the R language. Is there any method for accomplishing this in python? I can take this approach with the use of multiple 'equals' operators, but this is inconvenient when you want to consider a large number of items: df[(f.V1 == 'A') | (f.V1 == 'B'),:]  datatable 0.10.1 python 3.6 

Answer: You could also try this out: First import all the necessary packages as, import datatable as dt from datatable import by,f,count import functools import operator  Create a sample datatable: DT = dt.Frame(V1=['A','B','C','D','E','B','A'], V2=[1,2,3,4,5,6,7])  Make a list of values to be filtered among the observations, in your case it is sel_obs = ['A','B']  Now create a filter expression using funtools and operators modules, filter_rows = functools.reduce(operator.or_,(f.V1==obs for obs in sel_obs))  Finally apply the above created filter on datatable DT[fil_rows,:]  its output as- Out[6]: | V1 V2 -- + -- -- 0 | A 1 1 | B 2 2 | B 6 3 | A 7 [4 rows x 2 columns]  You can just play around with operators to do different type of filterings. @sammyweemy's solution should also work.