eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: In the documentation of scipy, the 'frozen pdf', etc, is mentioned sometimes, but I don't know the meaning of it? Is it a statistical concept or scipy terminology? 

Answer: I agree that the docs are somewhat unclear on the issue. It seems that the frozen distribution fixes the first n moments for programmer's convenience. I am unaware of the term "forzen distribution" outside of SciPy. SciPy's frozen distribution is perhaps best described here:  Passing the loc and scale keywords time and again can become quite bothersome. The concept of freezing a RV is used to solve such problems. rv = gamma(1, scale=2.) By using rv we no longer have to include the scale or the shape parameters anymore. Thus, distributions can be used in one of two ways, either by passing all distribution parameters to each method call (such as we did earlier) or by freezing the parameters for the instance of the distribution. Let us check this: rv.mean(), rv.std() (2.0, 2.0) This is, indeed, what we should get.  In the scipy tutorial page, we see the following line:  (We explain the meaning of a frozen distribution below).  The only mention of frozen distribution after that point is the following:  The main additional methods of the not frozen distribution are related to the estimation of distribution parameters: fit: maximum likelihood estimation of distribution parameters, including location and scale fit_loc_scale: estimation of location and scale when shape parameters are given nnlf: negative log likelihood function expect: calculate the expectation of a function against the pdf or pmf
Question: I've stored my training and validation data on two separate TFRecord files, in which I store 4 values: signal A (float32 shape (150,)), signal B (float32 shape (150,)), label (scalar int64), id (string). My parsing function for reading is: def _parse_data_function(sample_proto): raw_signal_description = { 'label': tf.io.FixedLenFeature([], tf.int64), 'id': tf.io.FixedLenFeature([], tf.string), } for key, item in SIGNALS.items(): raw_signal_description[key] = tf.io.FixedLenFeature(item, tf.float32) # Parse the input tf.Example proto using the dictionary above. return tf.io.parse_single_example(sample_proto, raw_signal_description)  where SIGNALS is a dictionary mapping signal name->signal shape. Then, I read the raw datasets: training_raw = tf.data.TFRecordDataset(<path to training>), compression_type='GZIP') val_raw = tf.data.TFRecordDataset(<path to validation>), compression_type='GZIP')  and use map to parse the values: training_data = training_raw.map(_parse_data_function) val_data = val_raw.map(_parse_data_function)  Displaying the header of training_data or val_data, I get: <MapDataset shapes: {Signal A: (150,), Signal B: (150,), id: (), label: ()}, types: {Signal A: tf.float32, Signal B: tf.float32, id: tf.string, label: tf.int64}> which is pretty much as expected. I also checked some of the values for consistency and they seemed to be correct. Now, to my issue: how do I get from the MapDataset, with the dictionary like structure, to something that can be given as input to the model? The input to my model is the pair (Signal A, label), though in the future I will use Signal B as well.  The simplest way to me seemed to create an generator over the elements that I want. Something like: def data_generator(mapdataset): for sample in mapdataset: yield (sample['Signal A'], sample['label'])  However, with this approach I lose some convenience of Datasets, such as batching, and it is also not clear how to use the same approach for the validation_data paramenter of model.fit. Ideally, I would only convert between the map representation and the Dataset representation where it iterates over pairs of Signal A tensors and labels. EDIT: My end product should be something with a header akin to: <TensorSliceDataset shapes: ((150,), ()), types: (tf.float32, tf.int64)> But not necessarily TensorSliceDataset

Answer: You can simply do this in the parse function. For example: def _parse_data_function(sample_proto): raw_signal_description = { 'label': tf.io.FixedLenFeature([], tf.int64), 'id': tf.io.FixedLenFeature([], tf.string), } for key, item in SIGNALS.items(): raw_signal_description[key] = tf.io.FixedLenFeature(item, tf.float32) # Parse the input tf.Example proto using the dictionary above. parsed = tf.io.parse_single_example(sample_proto, raw_signal_description) return parsed['Signal A'], parsed['label']  If you map this function over the TFRecordDataset, you will have a dataset of tuples (signal_a, label) instead of a dataset of dictionaries. You should be able to put this into model.fit directly.
Question: This is Python 3.7 I have a dataclass like this: @dataclass class Action: action: str  But action is actually restricted to the values "bla" and "foo". Is there a sensible way to express this?

Answer: You could use an Enum: from dataclasses import dataclass from enum import Enum class ActionType(Enum): BLA = 'bla' FOO = 'foo' @dataclass class Action: action: ActionType >>> a = Action(action=ActionType.FOO) >>> a.action.name 'FOO' >>> a.action.value 'foo'
Question: Here is a variable html_str, it is a string that contains html tags and contents in body. I am created a .html file from this string using the below code in python. html_file = open("filename.html", "w") html_file.write(html_str) html_file.close()  now i got html file named file "filename.html". Now i want to convert that "filename.html" to a image, named filename.jpg with the exact contents of the html file. please help me.

Answer: You can do this by using imgkit import imgkit imgkit.from_file('test.html', 'out.jpg')  Or you can also use htmlcsstoimage Api # pip3 install requests import requests HCTI_API_ENDPOINT = "https://hcti.io/v1/image" HCTI_API_USER_ID = 'your-user-id' HCTI_API_KEY = 'your-api-key' data = { 'html': "<div class='box'>Hello, world!</div>", 'css': ".box { color: white; background-color: #0f79b9; padding: 10px; font-family: Roboto }", 'google_fonts': "Roboto" } image = requests.post(url = HCTI_API_ENDPOINT, data = data, auth=(HCTI_API_USER_ID, HCTI_API_KEY)) print("Your image URL is: %s"%image.json()['url']) # https://hcti.io/v1/image/7ed741b8-f012-431e-8282-7eedb9910b32
Question: I want to train a model in a sequential manner. That is I want to train the model initially with a simple architecture and once it is trained, I want to add a couple of layers and continue training. Is it possible to do this in Keras? If so, how?  I tried to modify the model architecture. But until I compile, the changes are not effective. Once I compile, all the weights are re-initialized and I lose all the trained information. All the questions in web and SO I found are either about loading a pre-trained model and continuing training or modifying the architecture of pre-trained model and then only test it. I didn't find anything related to my question. Any pointers are also highly appreciated. PS: I'm using Keras in tensorflow 2.0 package.

Answer: Without knowing the details of your model, the following snippet might help: from tensorflow.keras.models import Model from tensorflow.keras.layers import Dense, Input # Train your initial model def get_initial_model(): ... return model model = get_initial_model() model.fit(...) model.save_weights('initial_model_weights.h5') # Use Model API to create another model, built on your initial model initial_model = get_initial_model() initial_model.load_weights('initial_model_weights.h5') nn_input = Input(...) x = initial_model(nn_input) x = Dense(...)(x) # This is the additional layer, connected to your initial model nn_output = Dense(...)(x) # Combine your model full_model = Model(inputs=nn_input, outputs=nn_output) # Compile and train as usual full_model.compile(...) full_model.fit(...)  Basically, you train your initial model, save it. And reload it again, and wrap it together with your additional layers using the Model API. If you are not familiar with Model API, you can check out the Keras documentation here (afaik the API remains the same for Tensorflow.Keras 2.0). Note that you need to check if your initial model's final layer's output shape is compatible with the additional layers (e.g. you might want to remove the final Dense layer from your initial model if you are just doing feature extraction).
Question: I am trying to implement Tacotron speech synthesis with Tensorflow in Google Colab using this code form a repo in Github, below is my code and working good till the step of using localhost server, how I can to run a localhost server in a notebook in Google Colab? My code:  !pip install tensorflow==1.3.0 import tensorflow as tf print("You are using Tensorflow",tf.__version__) !git clone https://github.com/keithito/tacotron.git cd tacotron pip install -r requirements.txt !curl https://data.keithito.com/data/speech/tacotron-20180906.tar.gz | tar xzC /tmp !python demo_server.py --checkpoint /tmp/tacotron-20180906/model.ckpt #requires localhost  Unfortunately running in local mode from Google Colab will not help me because to do this I need to download the data in my machine which are too large. Below is my last output and here I am supposed to open the localhost:8888 to complete the work, so as I mentioned before is there any way to run localhost in Google Colaboratory? 

Answer: You can do this by using tools like ngrok or remote.it They give you a URL that you can access from any browser to access your web server running on 8888 Example 1: Tunneling tensorboard running on  !wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip !unzip ngrok-stable-linux-amd64.zip get_ipython().system_raw('tensorboard --logdir /content/trainingdata/objectdetection/ckpt_output/trainingImatges/ --host 0.0.0.0 --port 6006 &') get_ipython().system_raw('./ngrok http 6006 &') ! curl -s http://localhost:4040/api/tunnels | python3 -c \ "import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])"  Running this install ngrok on colab, and makes a link like http://c11e1b53.ngrok.io/ Documentaion for NGROK
Question: I got the output but trying to find a more efficient way to do this:  (df['budget'] == 0).sum(), (df['revenue'] == 0).sum(),(df['budget_adj'] == 0).sum(), (df['revenue_adj'] == 0).sum()  Output is  (5674, 5993, 5676, 5993) 

Answer: You can compare the columns in bulk and sum these up column-wise: (df[['budget', 'revenue', 'budget_adj', 'revenue_adj']] == 0).sum(axis=0)
Question: Apparently list(a) doesn't overallocate, [x for x in a] overallocates at some points, and [*a] overallocates all the time?  Here are sizes n from 0 to 12 and the resulting sizes in bytes for the three methods: 0 56 56 56 1 64 88 88 2 72 88 96 3 80 88 104 4 88 88 112 5 96 120 120 6 104 120 128 7 112 120 136 8 120 120 152 9 128 184 184 10 136 184 192 11 144 184 200 12 152 184 208  Computed like this, reproducable at repl.it, using Python 3.8: from sys import getsizeof for n in range(13): a = [None] * n print(n, getsizeof(list(a)), getsizeof([x for x in a]), getsizeof([*a]))  So: How does this work? How does [*a] overallocate? Actually, what mechanism does it use to create the result list from the given input? Does it use an iterator over a and use something like list.append? Where is the source code? (Colab with data and code that produced the images.) Zooming in to smaller n:  Zooming out to larger n: 

Answer: [*a] is internally doing the C equivalent of:  Make a new, empty list Call newlist.extend(a) Returns list.  So if you expand your test to: from sys import getsizeof for n in range(13): a = [None] * n l = [] l.extend(a) print(n, getsizeof(list(a)), getsizeof([x for x in a]), getsizeof([*a]), getsizeof(l))  Try it online! you'll see the results for getsizeof([*a]) and l = []; l.extend(a); getsizeof(l) are the same. This is usually the right thing to do; when extending you're usually expecting to add more later, and similarly for generalized unpacking, it's assumed that multiple things will be added one after the other. [*a] is not the normal case; Python assumes there are multiple items or iterables being added to the list ([*a, b, c, *d]), so overallocation saves work in the common case. By contrast, a list constructed from a single, presized iterable (with list()) may not grow or shrink during use, and overallocating is premature until proven otherwise; Python recently fixed a bug that made the constructor overallocate even for inputs with known size. As for list comprehensions, they're effectively equivalent to repeated appends, so you're seeing the final result of the normal overallocation growth pattern when adding an element at a time. To be clear, none of this is a language guarantee. It's just how CPython implements it. The Python language spec is generally unconcerned with specific growth patterns in list (aside from guaranteeing amortized O(1) appends and pops from the end). As noted in the comments, the specific implementation changes again in 3.9; while it won't affect [*a], it could affect other cases where what used to be "build a temporary tuple of individual items and then extend with the tuple" now becomes multiple applications of LIST_APPEND, which can change when the overallocation occurs and what numbers go into the calculation.
Question: How can I calculate the Euclidean distance between all the rows of a dataframe? I am trying this code, but it is not working: zero_data = data distance = lambda column1, column2: pd.np.linalg.norm(column1 - column2) result = zero_data.apply(lambda col1: zero_data.apply(lambda col2: distance(col1, col2))) result.head()  This is how my (44062 by 278) dataframe looks like: 

Answer: To compute the Eucledian distance between two rows i and j of a dataframe df: np.linalg.norm(df.loc[i] - df.loc[j])  To compute it between consecutive rows, i.e. 0 and 1, 1 and 2, 2 and 3, ... np.linalg.norm(df.diff(axis=0).drop(0), axis=1)  If you want to compute it between all the rows, i.e. 0 and 1, 0 and 2, ..., 1 and 1, 1 and 2 ..., then you have to loop through all the combinations of i and j (keep in mind that for 44062 rows there are 970707891 such combinations so using a for-loop will be very slow): import itertools for i, j in itertools.combinations(df.index, 2): d_ij = np.linalg.norm(df.loc[i] - df.loc[j])  Edit: Instead, you can use scipy.spatial.distance.cdist which computes distance between each pair of two collections of inputs: from scipy.spatial.distance import cdist cdist(df, df, 'euclid')  This will return you a symmetric (44062 by 44062) matrix of Euclidian distances between all the rows of your dataframe. The problem is that you need a lot of memory for it to work (at least 8*44062**2 bytes of memory, i.e. ~16GB). So a better option is to use pdist from scipy.spatial.distance import pdist pdist(df.values, 'euclid')  which will return an array (of size 970707891) of all the pairwise Euclidean distances between the rows of df. P.s. Don't forget to ignore the 'Actual_data' column in the computations of distances. E.g. you can do the following: data = df.drop('Actual_Data', axis=1).values and then cdist(data, data, 'euclid') or pdist(data, 'euclid'). You can also create another dataframe with distances like this: data = df.drop('Actual_Data', axis=1).values d = pd.DataFrame(itertools.combinations(df.index, 2), columns=['i','j']) d['dist'] = pdist(data, 'euclid') i j dist 0 0 1 ... 1 0 2 ... 2 0 3 ... 3 0 4 ... ...
Question: After reading the answers to the question about monkey-patching classes in Python I tried to apply the advised solution to the following case. Imagine that we have a module a.py class A(object): def foo(self): print(1) class AA(A): pass  and let us try to monkey patch it as follows. It works when we monkey patch class A: >>> import a >>> class B(object): ... def foo(self): ... print(3) ... >>> a.A = B >>> x = a.A() >>> x.foo() 3  But if we try the inherited class, it turns to be not patched: >>> y = a.AA() >>> y.foo() 1  Is there any way to monkey patch the class with all its inherited classes? EDIT For now, the best solution for me is as follows: >>> class AB(B, a.AA): ... pass ... >>> a.AA = AB >>> x = a.AA() >>> x.foo() 3  Any complex structure of a.AA will be inherited and the only difference between AB and a.AA will be the foo() method. In this way, we don't modify any internal class attributes (like __base__ or __dict__). The only remaining drawback is that we need to do that for each of the inherited classes. Is it the best way to do this?

Answer: You need to explicitly overwrite the tuple of base classes in a.AA, though I don't recommend modifying classes like this. >>> import a >>> class B: ... def foo(self): ... print(2) ... >>> a.AA.__bases__ = (B,) >>> a.AA().foo() 2  This will also be reflected in a.A.__subclasses__() (although I am not entirely sure as to how that works; the fact that it is a method suggests that it computes this somehow at runtime, rather than simply returning a value that was modified by the original definition of AA). It appears that the bases classes in a class statement are simply remembered, rather than used, until some operation needs them (e.g. during attribute lookup). There may be some other subtle corner cases that aren't handled as smoothly: caveat programmator.
Question: When we import a module that isn't currently installed on the Python used on the current environment, PyCharm suggest us to 'install missing module', if you click install, it'll install it automatically... Is there any plugin for vscode that does that or something like that? I want to import emoji for example, and like pycharm does, suggest me to install the missing module, so I won't have to do manually a pip install Is there a plugin that does this for vscode? Thank you

Answer: I think that your request is already ongoing on vscode-python extension: https://github.com/microsoft/vscode-python/issues/8062 I suggest to follow this issue to see when it's ready for production.
Question: Keras allows for saving entire models or just model weights (see thread). When saving the weights, they must be saved to a file, eg: model = keras_model() model.save_weights('/tmp/model.h5')  Instead of writing to file, I'd like to just save the bytes into memory. Something like model.dump_weights()  Tensorflow doesn't seem to have this, so as a workaround I'm writing to disk and then reading into memory: temp = '/tmp/weights.h5' model.save_weights(temp) with open(temp, 'rb') as f: weightbytes = f.read()  Any way to avoid this roundabout? 

Answer: Thanks @ddoGas for pointing out the model.get_weights() method, which returns a list of weights that can then be serialized. Just some context for why I am not saving the model in the conventional way: we are working with model wrapper classes that associate a model and custom behavior. For example, before prediction occurs special validation is needed: class CNN: ... def predict(): self.do_special_validation() self.model.predict()  Hence, we're serializing the CNN class not just the underlying model. This is the solution to pickle the entire object. (pickle(CNN()) fails, otherwise we'd just use that) import pickle def serialize(cnn): return pickle.dumps({ "weights": cnn.model.get_weights(), "cnnclass": cnn.__class__ }) def deserialize(cnn_bytes): loaded = pickle.loads(cnn_bytes) weights, cnnclass = loaded['weights'], loaded['cnnclass'] cnninstance = cnnclass() cnninstance.model.set_weights(weights) return cnninstance  Works well, thanks! PS note using cnn.__class__ because don't want to necessarily bind this to the CNN class directly but for it to work in general for any class that has a cnn.model attribute.
Question: With a list containing some missing values such as this: [10, 11, 12,np.nan, 14, np.nan, 16, 17, np.nan, 19, np.nan]  How can you subset the values that are positioned between two missing (nan) values? I know how to do it with a for loop : # imports import numpy as np # input lst=[10,11,12,np.nan, 14, np.nan, 16, 17, np.nan, 19, np.nan] # define an empty list and build on that in a For Loop subset=[] for i, elem in enumerate(lst): if np.isnan(lst[i-1]) and np.isnan(lst[i+1]): subset.extend([elem]) print(subset) # output # [14, 19]  Any suggestions on how to do this in a less cumbersome way?

Answer: Use list comprehension import numpy as np lst=[10,11,12,np.nan, 14, np.nan, 16, 17, np.nan, np.nan, np.nan] subset = [elem for i, elem in enumerate(lst) if i and i < len(lst)-1 and np.isnan(lst[i-1]) and np.isnan(lst[i+1]) and not np.isnan(elem)] print(subset)  Corrected the mistakes that were pointed out by other contributors. This should work for all the cases now.
Question: Let's say I have a NumPy array: x = np.array([0, 1, 2, 0, 4, 5, 6, 7, 0, 0])  At each index, I want to find the distance to nearest zero value. If the position is a zero itself then return zero as a distance. Afterward, we are only interested in distances to the nearest zero that is to the right of the current position. The super naive approach would be something like: out = np.full(x.shape[0], x.shape[0]-1) for i in range(x.shape[0]): j = 0 while i + j < x.shape[0]: if x[i+j] == 0: break j += 1 out[i] = j  And the output would be: array([0, 2, 1, 0, 4, 3, 2, 1, 0, 0])  I'm noticing a countdown/decrement pattern in the output in between the zeros. So, I might be able to do use the locations of the zeros (i.e., zero_indices = np.argwhere(x == 0).flatten()) What is the fastest way to get the desired output in linear time? 

Answer: Approach #1 : Searchsorted to the rescue for linear-time in a vectorized manner (before numba guys come in)! mask_z = x==0 idx_z = np.flatnonzero(mask_z) idx_nz = np.flatnonzero(~mask_z) # Cover for the case when there's no 0 left to the right # (for same results as with posted loop-based solution) if x[-1]!=0: idx_z = np.r_[idx_z,len(x)] out = np.zeros(len(x), dtype=int) idx = np.searchsorted(idx_z, idx_nz) out[~mask_z] = idx_z[idx] - idx_nz  Approach #2 : Another with some cumsum - mask_z = x==0 idx_z = np.flatnonzero(mask_z) # Cover for the case when there's no 0 left to the right if x[-1]!=0: idx_z = np.r_[idx_z,len(x)] out = idx_z[np.r_[False,mask_z[:-1]].cumsum()] - np.arange(len(x))  Alternatively, last step of cumsum could be replaced by repeat functionality - r = np.r_[idx_z[0]+1,np.diff(idx_z)] out = np.repeat(idx_z,r)[:len(x)] - np.arange(len(x))  Approach #3 : Another with mostly just cumsum - mask_z = x==0 idx_z = np.flatnonzero(mask_z) pp = np.full(len(x), -1) pp[idx_z[:-1]] = np.diff(idx_z) - 1 if idx_z[0]==0: pp[0] = idx_z[1] else: pp[0] = idx_z[0] out = pp.cumsum() # Handle boundary case and assigns 0s at original 0s places out[idx_z[-1]:] = np.arange(len(x)-idx_z[-1],0,-1) out[mask_z] = 0