eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm using the bulk_create method from Django to create many entries at once. To ensure that the changes are only committed if there is no exception I'm thinking about adding transaction.atomic() to the code blocks but I'm not sure if I need to add it. From my understanding I only need to add it in Scenario 2 because in this case I'm executing more than one query. Scenario 1 Create 1.000 entries in one query Entry.objects.bulk_create([ Entry(headline='This is a test'), Entry(headline='This is only a test'), # ... ])  Scenario 2 Create 10.000 entries in in batches of 1.000 Entry.objects.bulk_create([ Entry(headline='This is a test'), Entry(headline='This is only a test'), # ... ], batch_size=1_000) 

Answer: No, you don't have to for either scenario. According to the Django source code, using transaction atomic would be redundant for bulk_create as that method already uses atomic transactions.
Question: I'm trying to interact with the page "Your connection is not private". The solution of using options.add_argument('--ignore-certificate-errors') is not helpful for two reasons:  I'm using an already open window. Even if I was using a "selenium opened window" the script runs non stop, and the issue I'm trying to solve is when my browser disconnects from a splunk dashboard and I want it to automatically connect again(and it pops the private connection window).  How do I click on "Advanced" and then click on "Proceed to splunk_server (unsafe)?

Answer: For chrome: from selenium import webdriver options = webdriver.ChromeOptions() options.add_argument('--ignore-ssl-errors=yes') options.add_argument('--ignore-certificate-errors') driver = webdriver.Chrome(options=options)  If not work then this: from selenium import webdriver from selenium.webdriver import DesiredCapabilities options = webdriver.ChromeOptions() options.add_argument('--allow-insecure-localhost') # differ on driver version. can ignore. caps = options.to_capabilities() caps["acceptInsecureCerts"] = True driver = webdriver.Chrome(desired_capabilities=caps)  For firefox: from selenium import webdriver profile = webdriver.FirefoxProfile() profile.accept_untrusted_certs = True driver = webdriver.Firefox(firefox_profile=profile) driver.get('https://cacert.org/') driver.close()  If not work then this: capabilities = webdriver.DesiredCapabilities().FIREFOX capabilities['acceptSslCerts'] = True driver = webdriver.Firefox(capabilities=capabilities) driver.get('https://cacert.org/') driver.close()  Above all worked for me!
Question: I have to convert some files which come by default as .dcm to .png, I've found some code samples to achieve that around here but the end results are too bright. Could anybody have a look at this, please?  def convert_to_png(file): ds = pydicom.dcmread(file) shape = ds.pixel_array.shape # Convert to float to avoid overflow or underflow losses. image_2d = ds.pixel_array.astype(float) # Rescaling grey scale between 0-255 image_2d_scaled = (np.maximum(image_2d,0) / image_2d.max()) * 255.0 # Convert to uint image_2d_scaled = np.uint8(image_2d_scaled) # Write the PNG file with open(f'{file.strip(".dcm")}.png', 'wb') as png_file: w = png.Writer(shape[1], shape[0], greyscale=True) w.write(png_file, image_2d_scaled)  I've tweaked around the code but nothing seems to work.  This is how the actual thing looks like as dicom and on the right side is the result of running this code 

Answer: Some DICOM datasets require window center/width rescaling of the original pixel intensities (via the (0028,1050) Window Center and (0028,1051) Window Width elements in the VOI LUT Module) in order to reproduce the way they were "viewed". pydicom has a function apply_voi_lut() for applying this windowing: from pydicom import dcmread from pydicom.pixel_data_handlers.util import apply_voi_lut ds = dcmread(file) if 'WindowWidth' in ds: print('Dataset has windowing') windowed = apply_voi_lut(ds.pixel_array, ds) # Add code for rescaling to 8-bit...  Depending on the dataset type you may need to use apply_modality_lut() beforehand.
Question: Is there any library to perform a Multivariate Multiple Regression (a Multiple Regression with multiple dependent variables) in Python? Greetings and thanks in advance

Answer: You can try the modules in sklearn, the response variable can be 2 or more dimensional, and i think it works for OLS (linear regression), lasso, ridge.. The models in statsmodels can only do 1 response (just checked). Example dataset: import pandas as pd from sklearn.datasets import load_iris iris = load_iris() df = pd.DataFrame(data= iris['data'], columns= iris['feature_names'] ) df.shape (150, 4)  Now we do the fit: from sklearn import linear_model clf = linear_model.LinearRegression() clf.fit(df[['sepal length (cm)']],df[['petal length (cm)','petal width (cm)']]) clf.coef_ array([[1.85843298], [0.75291757]])  You can see the coefficients are the same as when you fit one response in this case: clf.fit(df[['sepal length (cm)']],df[['petal width (cm)']]) clf.coef_ array([[0.75291757]])
Question: I'm trying to compare two data frames with have same number of columns i.e. 4 columns with id as key column in both data frames df1 = spark.read.csv("/path/to/data1.csv") df2 = spark.read.csv("/path/to/data2.csv")  Now I want to append new column to DF2 i.e. column_names which is the list of the columns with different values than df1 df2.withColumn("column_names",udf())  DF1 +------+---------+--------+------+ | id | |name | sal | Address | +------+---------+--------+------+ | 1| ABC | 5000 | US | | 2| DEF | 4000 | UK | | 3| GHI | 3000 | JPN | | 4| JKL | 4500 | CHN | +------+---------+--------+------+  DF2: +------+---------+--------+------+ | id | |name | sal | Address | +------+---------+--------+------+ | 1| ABC | 5000 | US | | 2| DEF | 4000 | CAN | | 3| GHI | 3500 | JPN | | 4| JKL_M | 4800 | CHN | +------+---------+--------+------+  Now I want DF3 DF3: +------+---------+--------+------+--------------+ | id | |name | sal | Address | column_names | +------+---------+--------+------+--------------+ | 1| ABC | 5000 | US | [] | | 2| DEF | 4000 | CAN | [address] | | 3| GHI | 3500 | JPN | [sal] | | 4| JKL_M | 4800 | CHN | [name,sal] | +------+---------+--------+------+--------------+  I saw this SO question, How to compare two dataframe and print columns that are different in scala. Tried that, however the result is different. I'm thinking of going with a UDF function by passing row from each dataframe to udf and compare column by column and return column list. However for that both the data frames should be in sorted order so that same id rows will be sent to udf. Sorting is costly operation here. Any solution?

Answer: Assuming that we can use id to join these two datasets I don't think that there is a need for UDF. This could be solved just by using inner join, array and array_remove functions among others. First let's create the two datasets: df1 = spark.createDataFrame([ [1, "ABC", 5000, "US"], [2, "DEF", 4000, "UK"], [3, "GHI", 3000, "JPN"], [4, "JKL", 4500, "CHN"] ], ["id", "name", "sal", "Address"]) df2 = spark.createDataFrame([ [1, "ABC", 5000, "US"], [2, "DEF", 4000, "CAN"], [3, "GHI", 3500, "JPN"], [4, "JKL_M", 4800, "CHN"] ], ["id", "name", "sal", "Address"])  First we do an inner join between the two datasets then we generate the condition df1[col] != df2[col] for each column except id. When the columns aren't equal we return the column name otherwise an empty string. The list of conditions will consist the items of an array from which finally we remove the empty items: from pyspark.sql.functions import col, array, when, array_remove # get conditions for all columns except id conditions_ = [when(df1[c]!=df2[c], lit(c)).otherwise("") for c in df1.columns if c != 'id'] select_expr =[ col("id"), *[df2[c] for c in df2.columns if c != 'id'], array_remove(array(*conditions_), "").alias("column_names") ] df1.join(df2, "id").select(*select_expr).show() # +---+-----+----+-------+------------+ # | id| name| sal|Address|column_names| # +---+-----+----+-------+------------+ # | 1| ABC|5000| US| []| # | 3| GHI|3500| JPN| [sal]| # | 2| DEF|4000| CAN| [Address]| # | 4|JKL_M|4800| CHN| [name, sal]| # +---+-----+----+-------+------------+
Question: I am developing a web-based application using Python, Flask, MySQL, and uWSGI. However, I am not using SQL Alchemy or any other ORM. I am working with a preexisting database from an old PHP application that wouldn't play well with an ORM anyway, so I'm just using mysql-connector and writing queries by hand. The application works correctly when I first start it up, but when I come back the next morning I find that it has become broken. I'll get errors like mysql.connector.errors.InterfaceError: 2013: Lost connection to MySQL server during query or the similar mysql.connector.errors.OperationalError: 2055: Lost connection to MySQL server at '10.0.0.25:3306', system error: 32 Broken pipe. I've been researching it and I think I know what the problem is. I just haven't been able to find a good solution. As best as I can figure, the problem is the fact that I am keeping a global reference to the database connection, and since the Flask application is always running on the server, eventually that connection expires and becomes invalid. I imagine it would be simple enough to just create a new connection for every query, but that seems like a far from ideal solution. I suppose I could also build some sort of connection caching mechanism that would close the old connection after an hour or so and then reopen it. That's the best option I've been able to come up with, but I still feel like there ought to be a better one. I've looked around, and most people that have been receiving these errors have huge or corrupted tables, or something to that effect. That is not the case here. The old PHP application still runs fine, the tables all have less than about 50,000 rows, and less than 30 columns, and the Python application runs fine until it has sat for about a day. So, here's to hoping someone has a good solution for keeping a continually open connection to a MySQL database. Or maybe I'm barking up the wrong tree entirely, if so hopefully someone knows.

Answer: I have it working now. Using pooled connections seemed to fix the issue for me. mysql.connector.connect( host='10.0.0.25', user='xxxxxxx', passwd='xxxxxxx', database='xxxxxxx', pool_name='batman', pool_size = 3 ) def connection(): """Get a connection and a cursor from the pool""" db = mysql.connector.connect(pool_name = 'batman') return (db, db.cursor())  I call connection() before each query function and then close the cursor and connection before returning. Seems to work. Still open to a better solution though. Edit I have since found a better solution. (I was still occasionally running into issues with the pooled connections). There is actually a dedicated library for Flask to handle mysql connections, which is almost a drop-in replacement. From bash: pip install Flask-MySQL Add MYSQL_DATABASE_HOST, MYSQL_DATABASE_USER, MYSQL_DATABASE_PASSWORD, MYSQL_DATABASE_DB to your Flask config. Then in the main Python file containing your Flask App object: from flaskext.mysql import MySQL mysql = MySQL() mysql.init_app(app)  And to get a connection: mysql.get_db().cursor() All other syntax is the same, and I have not had any issues since. Been using this solution for a long time now.
Question: Are there any caveats to it? I have a few questions related to it. How costly is it to create more GILs? Is it any different from creating a separate python runtime? Once a new GIL is created, will it create everything (objects, variables, stack, heap) from scratch as required in that process or a copy of everything in the present heap and the stack is created? (Garbage collection would malfunction if they are working on same objects.) Are the pieces of code being executed also copied to new CPU cores? Also can i relate one GIL to one CPU core? Now copying things is a fairly CPU intensive task (correct me if I am wrong), what would be the threshold to decide whether to go for multiprocessing? PS: I am talking about CPython but please feel free to extend the answer to whatever you feel is necessary.

Answer: Looking back at this question after 6 months, I feel I can clarify the doubts of my younger self. I hope this would be helpful to people who stumble upon it. Yes, It is true that in multiprocessing module, each process has a separate GIL and there are no caveats to it. But the understanding of the runtime and GIL is flawed in the question which needs to be corrected. I will clear the doubts/ answer the questions with a series of statements.  Python code is ran (compiled to Cpython bytecode and then this bytecode interpreted) by CPython virtual machine. This is what constitutes the python runtime. When we create a new process, an entire new python virtual machine is launched (which we call the python process) with the stack and the heap memory. Yes this is a costly process but not too costly. Because python virtual machine is piece of C code precompiled to machine code. To put in perspective, the reason that in java they do not use multiprocessing is that it will create multiple JVMs which would be terrible as JVM needs a lot of memory and also, JVM is not precompiled machine code like CPython. GIL is just a piece of code within the python virtual machine which lets the CPython interpreter execute only one line of CPython bytecode (or one instruction) at a time. So, all questions related to GIL creation and cost are dumb. Basically the intention was to ask about CPython Virtual Machine. Can I relate 1 GIL to 1 CPU core? : Better to ask if 1 Python process can be related to 1 CPU core? : No. That's Kernel's job to decide what core the process is running (and which will keep changing from time to time and the process would have no control over it). The only thing is that at any give point of time, one python process cannot be running on multiple cores and one python process will execute only one instruction in CPython bytecode (due to the GIL).  What's copied in cores and how the OS tries to keep a process hold the Core it is working on is a separate ans very deep topic in itself. The final question is a subjective one but with all this understanding, it's basically a cost to benefit ratio that may vary from program to program and might depend on how CPU intensive a process is and how many cores does the machine has etc. So that cannot be generalised.
Question: I installed Spark and when trying to run it, I am getting the error:  WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped  Can someone help me with that?

Answer: The same problem occured with me because python path was not added to system environment. I added this in environment and now it works perfectly. Adding PYTHONPATH environment variable with value as: %SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-<version>-src.zip;%PYTHONPATH%  helped resolve this issue. Just check what py4j version you have in your spark/python/lib folder.
Question: My distributed application consists of many producers that push tasks into several FIFO queues, and multiple consumers for every one of these queues. All these components live on a single node, so no networking involved. This pattern is perfectly supported by Python's built-in multiprocessing.Queue, however when I am scaling up my application the queue implementation seems to be a bottleneck. I am not sending large amounts of data, so memory sharing does not solve the problem. What I need is fast guaranteed delivery of 10^4-10^5 small messages per second. Each message is about 100 bytes. I am new to the world of fast distributed computing and I am very confused by the sheer amount of options. There is RabbitMQ, Redis, Kafka, etc. ZeroMQ is a more focused and compact alternative, which also has successors such as nanomsg and nng. Also, implementing something like a many-to-many queue with a guaranteed delivery seems nontrivial without a broker. I would really appreciate if someone could point me to a "standard" way of doing something like this with one of the faster frameworks.

Answer: After trying a few available implementations and frameworks, I still could not find anything that would be suitable for my task. Either too slow or too heavy. To solve the issue my colleagues and I developed this: https://github.com/alex-petrenko/faster-fifo faster-fifo is a drop-in replacement for Python's multiprocessing.Queue and is significantly faster. In fact, it is up to 30x faster in the configurations I cared about (many producers, few consumers) because it additionally supports get_many() method on the consumer side. It is brokereless, lightweight, supports arbitrary many-to-many configurations, implemented for Posix systems using pthread synchronization primitives.
Question: I am trying to implement a custom dataset for my neural network. But got this error when running the forward function. The code is as follows. import torch import torch.nn as nn import torch.nn.functional as F from torch.utils.data import Dataset, DataLoader import numpy as np class ParamData(Dataset): def __init__(self,file_name): self.data = torch.Tensor(np.loadtxt(file_name,delimiter = ',')) #first place def __len__(self): return self.data.size()[0] def __getitem__(self,i): return self.data[i] class Net(nn.Module): def __init__(self,in_size,out_size,layer_size=200): super(Net,self).__init__() self.layer = nn.Linear(in_size,layer_size) self.out_layer = nn.Linear(layer_size,out_size) def forward(self,x): x = F.relu(self.layer(x)) x = self.out_layer(x) return x datafile = 'data1.txt' net = Net(100,1) dataset = ParamData(datafile) n_samples = len(dataset) #dataset = torch.Tensor(dataset,dtype=torch.double) #second place #net.float() #thrid place net.forward(dataset[0]) #fourth place  In the file data1.txt is a csv formatted text file containing certain numbers, and each dataset[i] is a size 100 by 1 torch.Tensor object of dtype torch.float64. The error message is as follows: Traceback (most recent call last): File "Z:\Wrong.py", line 33, in <module> net.forward(dataset[0]) File "Z:\Wrong.py", line 23, in forward x = F.relu(self.layer(x)) File "E:\Python38\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "E:\Python38\lib\site-packages\torch\nn\modules\linear.py", line 87, in forward return F.linear(input, self.weight, self.bias) File "E:\Python38\lib\site-packages\torch\nn\functional.py", line 1372, in linear output = input.matmul(weight.t()) RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2' in call to _th_mm  It seems that I should change the dtype of the numbers in dataset to torch.double. I tried things like  changing the line at the first place to self.data = torch.tensor(np.loadtxt(file_name,delimiter = ','),dtype=torch.double) changing the line at the fourth place to net.forward(dataset[0].double()) uncommenting one of the two lines at the second or the thrid place  I think these are the solutions I have seen from similar questions, but they either give new errors or don't do anything. What should I do?  Update: So I got it working by changing the first place to self.data = torch.from_numpy(np.loadtxt(file_name,delimiter = ',')).float()  which is weird because it is exactly the opposite of the error message. Is this a bug? I'd still like some explaining.

Answer: Now that I have more experience with pytorch, I think I can explain the error message. It seems that the line RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2' in call to _th_mm  is actually refering to the weights of the linear layer when the matrix multiplication is called. Since the input is double while the weights are float, it makes sense for the line output = input.matmul(weight.t())  to expect the weights to be double.
Question: I want to use a numba.typed.List (going to call it List) to pass into my function which is wrapped in njit. However this List should be created from an existing python list. When I look at the documentation it seems the way you create a List is to initialize it and then append elements to it. However this requires you to loop over an already existing list in python which seems inefficient for large lists. For example: from numba.typed import List numba_list = List() py_list = ["a", "b", "c"] for e in py_list: numba_list.append(e) In [17]: numba_list[0] Out[17]: 'a'   Is there a way to set a List to the values of a python list without explicitly looping over the python list ?  I am using numba.__version__ = '0.47.0'

Answer: I am working on numba 0.49.1 where you can pass the list by the construction. py_list = [2,3,5] number_list = numba.typed.List(py_list)
Question: I am using python to extract Arabic tweets from twitter and save it as a CSV file, but when I open the saved file in excel the Arabic language displays as symbols. However, inside python, notepad, or word, it looks good. May I know where is the problem?

Answer: This is a problem I face frequently with Microsoft Excel when opening CSV files that contain Arabic characters. Try the following workaround that I tested on latest versions of Microsoft Excel on both Windows and MacOS:  Open Excel on a blank workbook Within the Data tab, click on From Text button (if not activated, make sure an empty cell is selected) Browse and select the CSV file In the Text Import Wizard, change the File_origin to "Unicode (UTF-8)" Go next and from the Delimiters, select the delimiter used in your file e.g. comma Finish and select where to import the data  The Arabic characters should show correctly.
Question: I was trying to plot a confusion matrix nicely, so I followed scikit-learn's newer version 0.22's in built plot confusion matrix function. However, one value of my confusion matrix value is 153, but it appears as 1.5e+02 in the confusion matrix plot:  Following the scikit-learn's documentation, I spotted this parameter called values_format, but I do not know how to manipulate this parameter so that it can suppress the scientific notation. My code is as follows. from sklearn import svm, datasets from sklearn.model_selection import train_test_split from sklearn.metrics import plot_confusion_matrix # import some data to play with X = pd.read_csv("datasets/X.csv") y = pd.read_csv("datasets/y.csv") class_names = ['Not Fraud (positive)', 'Fraud (negative)'] # Split the data into a training set and a test set X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42) # Run classifier, using a model that is too regularized (C too low) to see # the impact on the results logreg = LogisticRegression() logreg.fit(X_train, y_train) np.set_printoptions(precision=2) # Plot non-normalized confusion matrix titles_options = [("Confusion matrix, without normalization", None), ("Normalized confusion matrix", 'true')] for title, normalize in titles_options: disp = plot_confusion_matrix(logreg, X_test, y_test, display_labels=class_names, cmap=plt.cm.Greens, normalize=normalize, values_format = '{:.5f}'.format) disp.ax_.set_title(title) print(title) print(disp.confusion_matrix) plt.show() 

Answer: Just remove ".format" and the {} brackets from your call parameter declaration: disp = plot_confusion_matrix(logreg, X_test, y_test, display_labels=class_names, cmap=plt.cm.Greens, normalize=normalize, values_format = '.5f')  In addition, you can use '.5g' to avoid decimal 0's Taken from source
Question: I'm trying to read in a regular csv file into pandas through pd.read_csv(). I have done this on my local desktop many times before but I am using a virtual machine now and am getting this error : ImportError: cannot import name 'is_url' from 'pandas.io.common' (/opt/conda/lib/python3.7/site-packages/pandas/io/common.py)  Can anyone help me understand what's happening and how to resolve it? I have tried updating as well as uninstalling and reinstalling pandas already.

Answer: conda update --force-reinstall pandas  this worked for me.