eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am examining different methods in outlier detection. I came across sklearn's implementation of Isolation Forest and Amazon sagemaker's implementation of RRCF (Robust Random Cut Forest). Both are ensemble methods based on decision trees, aiming to isolate every single point. The more isolation steps there are, the more likely the point is to be an inlier, and the opposite is true. However, even after looking at the original papers of the algorithms, I am failing to understand exactly the difference between both algorithms. In what way do they work differently? Is one of them more efficient than the other? EDIT: I am adding the links to the research papers for more information, as well as some tutorials discussing the topics. Isolation Forest: Paper Tutorial Robust Random Cut Forest: Paper Tutorial

Answer: In part of my answers I'll assume you refer to Sklearn's Isolation Forest. I believe those are the 4 main differences:  Code availability: Isolation Forest has a popular open-source implementation in Scikit-Learn (sklearn.ensemble.IsolationForest), while both AWS implementation of Robust Random Cut Forest (RRCF) are closed-source, in Amazon Kinesis and Amazon SageMaker. There is an interesting third party RRCF open-source implementation on GitHub though: https://github.com/kLabUM/rrcf ; but unsure how popular it is yet  Training design: RRCF can work on streams, as highlighted in the paper and as exposed in the streaming analytics service Kinesis Data Analytics. On the other hand, the absence of partial_fit method hints me that Sklearn's Isolation Forest is a batch-only algorithm that cannot readily work on data streams  Scalability: SageMaker RRCF is more scalable. Sklearn's Isolation Forest is single-machine code, which can nonetheless be parallelized over CPUs with the n_jobs parameter. On the other hand, SageMaker RRCF can be used over one machine or multiple machines. Also, it supports SageMaker Pipe mode (streaming data via unix pipes) which makes it able to learn on much bigger data than what fits on disk  the way features are sampled at each recursive isolation: RRCF gives more weight to dimension with higher variance (according to SageMaker doc), while I think isolation forest samples at random, which is one reason why RRCF is expected to perform better in high-dimensional space (picture from the RRCF paper)
Question: I have two Pytorch tensors of the form [y11, y12] and [y21, y22]. How do I get the weighted mean of the two tensors?

Answer: you can add two tensors using torch.add and then get the mean of output tensor using torch.mean assuming weight as 0.6 for tensor1 and 0.4 for tensor2 example: tensor1 = [y11, y12] * 0.6 # multiplying with weight tensor2 = [y21, y22] * 0.4 # multiplying with weight pt_addition_result_ex = tensor1.add(tensor2) # addition of two tensors torch.mean(pt_addition_result_ex) # mean of output tensors
Question: I have a dataframe with this format: ID measurement_1 measurement_2 0 3 NaN 1 NaN 5 2 NaN 7 3 NaN NaN  I want to combine to: ID measurement measurement_type 0 3 1 1 5 2 2 7 2  For each row there will be a value in either measurement_1 or measurement_2 column, not in both, the other column will be NaN. In some rows both columns will be NaN. I want to add a column for the measurement type (depending on which column has the value) and take the actual value out of both columns, and remove the rows that have NaN in both columns. Is there an easy way of doing this? Thanks!

Answer: Use DataFrame.stack to reshape the dataframe then use reset_index and use DataFrame.assign to assign the column measurement_type by using Series.str.split + Series.str[:1] on level_1: df1 = ( df.set_index('ID').stack().reset_index(name='measurement') .assign(mesurement_type=lambda x: x.pop('level_1').str.split('_').str[-1]) )  Result: print(df1) ID measurement mesurement_type 0 0 3.0 1 1 1 5.0 2 2 2 7.0 2
Question: I'm using python to get time of 5 minutes ago. Code: import datetime import time now = datetime.datetime.now() current_time = now.strftime(f"%Y-%m-%d %H:%M") print(current_time) 2020-07-27 08:35:00  My question is how to get the time of 5 minutes ago. something like current_time-5mins 

Answer: You may use datetime.timedelta(): datetime.datetime.now() - datetime.timedelta(minutes=5)
Question: I am using the IDE called Spyder for learning Python. I would like to know in how to go about in installing Python packages for Spyder?

Answer: step 1. First open Spyder and click Tools --> Open command prompt.    For more details click visit this link, https://miamioh.instructure.com/courses/38817/pages/downloading-and-installing-packages
Question: I am new to pytorch, and i would like to know how to display graphs of loss and accuraccy And how exactly should i store these values,knowing that i'm applying a cnn model for image classification using CIFAR10. here is my current implementation :  def train(num_epochs,optimizer,criterion,model): for epoch in range(num_epochs): for i, (images, labels) in enumerate(trainloader): # origin shape: [4, 3, 32, 32] = 4, 3, 1024 # input_layer: 3 input channels, 6 output channels, 5 kernel size images = images.to(device) labels = labels.to(device) # Forward pass outputs = model(images) loss = criterion(outputs, labels) # Backward and optimize optimizer.zero_grad() loss.backward() optimizer.step() if (i+1) % 2000 == 0: print (f'Epoch [{epoch+1}/{num_epochs}], Step [{i+1}/{n_total_steps}], Loss: {loss.item():.4f}') PATH = './cnn.pth' torch.save(model.state_dict(), PATH) def test (): with torch.no_grad(): n_correct = 0 n_samples = 0 n_class_correct = [0 for i in range(10)] n_class_samples = [0 for i in range(10)] for images, labels in testloader: images = images.to(device) labels = labels.to(device) outputs = model(images) # max returns (value ,index) _, predicted = torch.max(outputs, 1) n_samples += labels.size(0) n_correct += (predicted == labels).sum().item() for i in range(batch_size): label = labels[i] pred = predicted[i] if (label == pred): n_class_correct[label] += 1 n_class_samples[label] += 1 acc = 100.0 * n_correct / n_samples print(f'Accuracy of the network: {acc} %') for i in range(10): acc = 100.0 * n_class_correct[i] / n_class_samples[i] print(f'Accuracy of {classes[i]}: {acc} %') test_score = np.mean([100 * n_class_correct[i] / n_class_samples[i] for i in range(10)]) print("the score test is : {0:.3f}%".format(test_score)) return acc 

Answer: What you need to do is: Average the loss over all the batches and then append it to a variable after every epoch and then plot it. Implementation would be something like this: import matplotlib.pyplot as plt def my_plot(epochs, loss): plt.plot(epochs, loss) def train(num_epochs,optimizer,criterion,model): loss_vals= [] for epoch in range(num_epochs): epoch_loss= [] for i, (images, labels) in enumerate(trainloader): # rest of the code loss.backward() epoch_loss.append(loss.item()) # rest of the code # rest of the code loss_vals.append(sum(epoch_loss)/len(epoch_loss)) # rest of the code # plotting my_plot(np.linspace(1, num_epochs, num_epochs).astype(int), loss_vals) my_plot([1, 2, 3, 4, 5], [100, 90, 60, 30, 10])  You can do a similar calculation for accuracy.
Question: I have a DataFrame: df = pd.DataFrame({'ID':['a','b','d','d','a','b','c','b','d','a','b','a'], 'sec':[3,6,2,0,4,7,10,19,40,3,1,2]}) print(df) ID sec 0 a 3 1 b 6 2 d 2 3 d 0 4 a 4 5 b 7 6 c 10 7 b 19 8 d 40 9 a 3 10 b 1 11 a 2  I want to calculate how many times a transition has occurred. Here in the ID column a->b is considered as a transition, similarly for b->d, d->d, d->a, b->c, c->b, b->a. I can do this using Counter like: Counter(zip(df['ID'].to_list(),df['ID'].to_list()[1:])) Counter({('a', 'b'): 3, ('b', 'd'): 2, ('d', 'd'): 1, ('d', 'a'): 2, ('b', 'c'): 1, ('c', 'b'): 1, ('b', 'a'): 1})  I also need to get min and max of the sec column of those transitions. Here for example a->b has occurred 3 times out of them min sec value is 1 and max sec value is 7. Also I want to get where this transition first occurred for a->b its 0. For the transition_index column I consider the first value of a transition, i.e. index of a and for calculating, min, max I take the second value of the transition, i.e. value at b. Here is the final output I want to get: df = pd.DataFrame({'ID_1':['a','b','d','d','b','c','b'], 'ID_2':['b','d','d','a','c','b','a'], 'sec_min':[1,2,0,3,10,19,2], 'sec_max':[7,40,0,4,10,19,2], 'transition_index':[0,1,2,3,5,6,10], 'count':[3,2,1,2,1,1,1]}) print(df) ID_1 ID_2 sec_min sec_max transition_index count 0 a b 1 7 0 3 1 b d 2 40 1 2 2 d d 0 0 2 1 3 d a 3 4 3 2 4 b c 10 10 5 1 5 c b 19 19 6 1 6 b a 2 2 10 1  How can I achieve this in Python? Also I have a huge amount of data, so I'm looking for the fastest way possible.

Answer: You have transitions of the form from -> to. 'transition_index' is based on the index of the "from" row, while the 'sec' aggregations are based on the value associated with the "to" row. We can shift the index and group on the ID and the shifted the ID, allowing us to use a single groupby with named aggregations to get the desired output.  df = df.reset_index() df['index'] = df['index'].shift().astype('Int64') (df.groupby([df['ID'].shift(1).rename('ID_1'), df['ID'].rename('ID_2')], sort=False) .agg(sec_min=('sec', 'min'), sec_max=('sec', 'max'), transition_index=('index', 'first'), count=('sec', 'size')) .reset_index() )    ID_1 ID_2 sec_min sec_max transition_index count 0 a b 1 7 0 3 1 b d 2 40 1 2 2 d d 0 0 2 1 3 d a 3 4 3 2 4 b c 10 10 5 1 5 c b 19 19 6 1 6 b a 2 2 10 1
Question: I firstly dump some stuff into a pickle file using pickle.dump. in utils.load_data, my project hierarchy looks like this project1 -utils -__init__.py -load_data.py -data (other folder...)  Then it outputs a pickle file into a data folder. Then I move the .pickle file to another project, the project hierarchy is project2 -data -main.py  When I run a pickle.load() operation in this main.py, it prompts the error as the title. However, if I move main.py back to project1 folder, then the error disappears. So the issue must be from the file. My question is, why does pickle try to import the package from where it was born? Could anyone share a good explanation for this? I got quite confused.

Answer: By default, unpickling will import any class that it finds in the pickle data. This means if you have pickled a custom class and you are unpickling it somewhere, pickle will try import the module (utils in this case). So you need to have the utils module inside project2 folder Follow this for more information
Question: I have the DataFrame df = pd.DataFrame({ 'colA':['?',2,3,4,'?'], 'colB':[1,2,'?',3,4], 'colC':['?',2,3,4,5] })  I would like to get the count the the number of '?' in each column and return the following output - colA - 2 colB - 1 colC - 1  Is there a way to return this output at once. Right now the only way I know how to do it is write a for loop for each column.

Answer: looks like the simple way is df[df == '?'].count() the result is colA 2 colB 1 colC 1 dtype: int64  where df[df == '?'] give us DataFrame with ? and Nan  colA colB colC 0 ? NaN ? 1 NaN NaN NaN 2 NaN ? NaN 3 NaN NaN NaN 4 ? NaN NaN  and the count non-NA cells for each column. Please, look on the other solutions: good readable and the most faster
Question: I use """Data taken from https://datos.gob.mx/busca/organization/conapo and https://es.wikipedia.org/wiki/Anexo:Entidades_federativas_de_M%C3%A9xico_por_superficie,_poblaci%C3%B3n_y_densidad """ total_population_segmentation = pd.read_html('professional_segmentation_mexico.html') population_segmentation = pd.read_html('population_segmentation.html')  followed by total_population_segmentation = population_segmentation[2] total_population_segmentation = total_population_segmentation['Población histórica de México'] total_population_segmentation = total_population_segmentation.drop('Pos',axis=1) total_population_segmentation = total_population_segmentation.sort_values('Entidad').reset_index().drop('index',axis=1)  Therefore, I am working with the following DataFrame  total_population_segmentation.head(5)   I used total_population_segmentation.dtypes and I got Entidad object 2010 object 2015 object 2020 object 2025 object 2030 object dtype: object  I used pd.to_numeric(total_population_segmentation['2010']) to check if it works but I got ValueError Traceback (most recent call last) pandas\_libs\lib.pyx in pandas._libs.lib.maybe_convert_numeric() ValueError: Unable to parse string "1 195 787" During handling of the above exception, another exception occurred: ValueError Traceback (most recent call last) <ipython-input-202-28db64f185e1> in <module>() ----> 1 pd.to_numeric(total_population_segmentation['2010']) ~\Anaconda3\lib\site-packages\pandas\core\tools\numeric.py in to_numeric(arg, errors, downcast) 148 try: 149 values = lib.maybe_convert_numeric( --> 150 values, set(), coerce_numeric=coerce_numeric 151 ) 152 except (ValueError, TypeError): pandas\_libs\lib.pyx in pandas._libs.lib.maybe_convert_numeric() ValueError: Unable to parse string "1 195 787" at position 0  When I look at each one of the values I obtain data that is decoded differently In [1]: total_population_segmentation['2010'][4] Out[1]: '4\xa0933\xa0755'  How can I convert this type of data to float?

Answer: it looks like you’ve got [NO-BREAK SPACE][1] character xa0 You should normalize your data first & convert it from string to integer. One way to do this is (this is just for one column) is like that: $ df = pd.DataFrame([ {'Entidad':'BajaCaliforniaSur', '2010': '3\xa0224\xa0884', '2015': '763\xa0321', '2030': '763\xa0321'}, {'Entidad':'BajaCaliforniaSur', '2010': '5\xa0224\xa0684', '2015': '763\xa0321', '2030': '763\xa0321'}, {'Entidad':'BajaCaforniaSur', '2010': '4\xa0214\xa0784' , '2015': '762\xa0321', '2030': '762\xa0321'}, {'Entidad':'BajaCaorniaSur', '2010': '8\xa0234\xa0684' , '2015': '761\xa0321', '2030': '761\xa0321'}, {'Entidad':'BajaCaorniaSur', '2010': '8\xa0234\xa0684' , '2015': '761\xa0321', '2030': '761\xa0321'}, {'Entidad':'BajaCalrniaSur', '2010': '2\xa0274\xa0084' , '2015': '769\xa0321', '2030': '769\xa0321'}]) $ from unidecode import unidecode $ df['2010'][0] '3\xa0224\xa0884' $ df['2010'] = df['2010'].apply(lambda x: (unidecode(x).replace(' ',''))).astype(float) $ df['2010'][0] 3224884.0
Question: As far as I understand Python destructors should be called when the reference count of an object reaches 0. But this assumption seems not to be correct. Look at the following code: class A: def __init__(self, b): self.__b = b print("Construct A") def __del__(self): # It seems that the destructor of B is called here. print("Delete A") # But it should be called here class B: def __init__(self): print("Construct B") def __del__(self): print("Delete B") b = B() a = A(b)  Outputs Construct B Construct A Delete B Delete A  But A has a reference to B, so I would expect the following output: Construct B Construct A Delete A Delete B  What am I not getting?

Answer: So, since the objects are still alive when the interpreter shuts down, you are actually not even guaranteed that __del__ will be called. At this point, the language makes no guarantees about when the finalizer is called. From the docs:  It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.  Note, if you change the script to: (py38) 173-11-109-137-SFBA:~ juan$ cat test.py class A: def __init__(self, b): self.__b = b print("Construct A") def __del__(self): # It seems that the destructor of B is called here. print("Delete A") # But it should be called here class B: def __init__(self): print("Construct B") def __del__(self): print("Delete B") b = B() a = A(b) del a del b  Then, executed: (py38) 173-11-109-137-SFBA:~ juan$ python test.py Construct B Construct A Delete A Delete B  Although del does not delete objects, it deletes references, so it forces the reference count to reach 0 while the interpreter is still running, so the order is as you would expect. Sometimes, __del__ won't be called at all. A common circumstance is file-objects created by f = open('test.txt')  That have live references in the global scope. If not closed explicitly, it might not call __del__ and the file will not flush and you won't get anything written. Which is a great reason to use the file object as a context-manager...
Question: I know how to drop duplicate rows based on column data. I also know how to drop dublicate rows based on row index. My question is: is there a way to drop duplicate rows based on index and one column? Thanks!

Answer: This can be done by turning the index into a column. Below is a sample data set (fyi, I think someone downvoted your question because it didn't include a sample data set): df=pd.DataFrame({'a':[1,2,2,3,4,4,5], 'b':[2,2,2,3,4,5,5]}, index=[0,1,1,2,3,5,5])  Output:  a b 0 1 2 1 2 2 1 2 2 2 3 3 3 4 4 5 4 5 5 5 5  Then you can use the following line. The first reset_index() makes a new column with the index numbers. Then you can drop duplicates based on the new index column and the other column (b in this case). Afterward, you can set the index to the original index values with set_index('index'): df.reset_index().drop_duplicates(subset=['index','b']).set_index('index')  Ouput:  a b index 0 1 2 1 2 2 2 3 3 3 4 4 5 4 5
Question: This is the last layer of a Keras model. model.add(Dense(3, activation='softmax')) model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])  I know that the output of the softmax layer is an array, with probability summing up to 1, such as [0.1, 0.4, 0.5]. I have one question about using accuracy as a metric. e.g., when the true class is [0, 0, 1] and predicted probability is [0.1, 0.4, 0.5], even if 0.5 is the largest probability, the accuracy of this prediction should be 0, because 0.5 != 1. Is that correct? More generally, when the output layer activation is softmax, we will normally get floating probability predictions, and in very very little chance will we get integer probability predictions like [0, 0, 1]. So we can't use accuracy as a metric when using softmax as activation. Is that correct?

Answer:  e.g., when the true class is [0, 0, 1] and predicted probability is [0.1, 0.4, 0.5], even if 0.5 is the largest probability, the accuracy of this prediction should be 0, because 0.5 != 1. Is that correct?  No. You treat the index with the maximum value as the prediction of the model. So in your example, this sample prediction would count towards increasing the accuracy. This is normally called Top-1 accuracy. In image classification, the Top-5 accuracy is also often used (the top 5 maximum values in the softmax layer are treated as guesses of the NN and they are considered for the accuracy).  More generally, when the output layer activation is softmax, we will normally get floating probability predictions, and in very very little chance will we get integer probability predictions like [0, 0, 1]. So we can't use accuracy as a metric when using softmax as activation. Is that correct?  Technically speaking, you will never get integer values for the softmax layer output since the type is float. But yeah, there's a very teeny tiny chance of getting [0.0, 0.0, 1.0]. And this assumption of yours is incorrect since the premise does not hold. Nevertheless, accuracy is a valid metric when using Softmax as the classification layer of a neural network.
Question: My PC is rocking a 2080TI so I don't really need the GPU computation of Google Colab, but I do find it a nice development environment (in comparison to jupyter notebooks) and I like the fact that I can access my files from anywhere, so, is it possible to use Google Colab but let my local pc do the computation?

Answer: Steps:  Go to Google colab and click on connect to local runtime, a pop-up comes.   Go to your terminal and execute:   jupyter notebook --NotebookApp.allow_origin='https://colab.research.google.com' --port=8888 --NotebookApp.port_retries=0  If this shows error: ERROR: the notebook server could not be started because port 8888 is not available.  Then run the following to basically kill any process using it or use another port:  lsof -wni tcp:8888 kill -9 <JOB_ID>  If successful, then the command gives a link: For example: http://localhost:8888/?token=bb80b05aef71999353fe4715e0f06be40d22911648dbdcd6  Copy it in the pop-up in the colab and you are set to go.
Question: I'm trying to adapt an example from Flask documentation to create a custom command in a group: import click from flask import Flask from flask.cli import AppGroup app = Flask(__name__) user_cli = AppGroup('user') @user_cli.command('create') @click.argument('name') def create_user(name): ... app.cli.add_command(user_cli) $ flask user create demo  This appears to work fine, however when I run flask --help I see the commands listed without any help messages, e.g.: Commands: user foo db Perform database migrations.  How can I add a help message to a group of commands ('user' in this case)?

Answer: Use the short_help parameter. AppGroup inherits from Group which inherits from MultiCommand which inherits from Command. See Click source code for Command. For example: import click from flask import Flask from flask.cli import AppGroup user_cli = AppGroup('user', short_help="Adds a user") @user_cli.command('create') @click.argument('name') def create_user(name): print(name) app = Flask(__name__) app.cli.add_command(user_cli) @app.route('/') def hello_world(): return 'Hello World!' if __name__ == '__main__': app.run()  Gives the following output (Windows PyCharm terminal): (flask_cli_group) D:\Paul\PycharmProjects\flask_cli_group>flask Usage: flask [OPTIONS] COMMAND [ARGS]... A general utility script for Flask applications. Provides commands from Flask, extensions, and the application. Loads the application defined in the FLASK_APP environment variable, or from a wsgi.py file. Setting the FLASK_ENV environment variable to 'development' will enable debug mode. > set FLASK_APP=hello.py > set FLASK_ENV=development > flask run Options: --version Show the flask version --help Show this message and exit. Commands: routes Show the routes for the app. run Run a development server. shell Run a shell in the app context. user Adds a user (flask_cli_group) D:\Paul\PycharmProjects\flask_cli_group>
Question: In Pytorch, I know that certain image processing transformations can be composed as such: import torchvision.transforms as transforms  transform = transforms.Compose([transforms.ToTensor(),  transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) In my case, each image has a corresponding annotation of bounding box coordinates with YOLO format. Does Pytorch allow to apply these transformations to the bounding box coordinates of the image as well, and later save them as new annotations? Thanks.

Answer: The transformations that you used as examples do not change the bounding box coordinates. ToTensor() converts a PIL image to a torch tensor and Normalize() is used to normalize the channels of the image. Transformations such as RandomCrop() and RandomRotation() will cause a mismatch between the location of the bounding box and the (modified) image. However, Pytorch makes it very flexible for you to create your own transformations and have control over what happens with the bounding box coordinates. Docs for more details: https://pytorch.org/docs/stable/torchvision/transforms.html#functional-transforms As an example (modified from the documentation): import torchvision.transforms.functional as TF import random def my_rotation(image, bonding_box_coordinate): if random.random() > 0.5: angle = random.randint(-30, 30) image = TF.rotate(image, angle) bonding_box_coordinate = TF.rotate(bonding_box_coordinate, angle) # more transforms ... return image, bonding_box_coordinate  Hope that helps =)