eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I am building a text classification model for imdb sentiment analysis dataset. I downloaded the dataset and followed the tutorial given here - https://developers.google.com/machine-learning/guides/text-classification/step-4 The error I get is TypeError: Failed to convert object of type <class 'tensorflow.python.framework.sparse_tensor.SparseTensor'> to Tensor. Contents: SparseTensor(indices=Tensor("DeserializeSparse:0", shape=(None, 2), dtype=int64), values=Tensor("DeserializeSparse:1", shape=(None,), dtype=float32), dense_shape=Tensor("stack:0", shape=(2,), dtype=int64)). Consider casting elements to a supported type.  the type of x_train and x_val are scipy.sparse.csr.csr_matrix. This give an error when passed to sequential model. How to solve? import tensorflow as tf import numpy as np from tensorflow.python.keras.preprocessing import sequence from tensorflow.python.keras.preprocessing import text from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.feature_selection import SelectKBest from sklearn.feature_selection import f_classif # Vectorization parameters # Range (inclusive) of n-gram sizes for tokenizing text. NGRAM_RANGE = (1, 2) # Limit on the number of features. We use the top 20K features. TOP_K = 20000 # Whether text should be split into word or character n-grams. # One of 'word', 'char'. TOKEN_MODE = 'word' # Minimum document/corpus frequency below which a token will be discarded. MIN_DOCUMENT_FREQUENCY = 2 # Limit on the length of text sequences. Sequences longer than this # will be truncated. MAX_SEQUENCE_LENGTH = 500 def ngram_vectorize(train_texts, train_labels, val_texts): """Vectorizes texts as ngram vectors. 1 text = 1 tf-idf vector the length of vocabulary of uni-grams + bi-grams. # Arguments train_texts: list, training text strings. train_labels: np.ndarray, training labels. val_texts: list, validation text strings. # Returns x_train, x_val: vectorized training and validation texts """ # Create keyword arguments to pass to the 'tf-idf' vectorizer. kwargs = { 'ngram_range': NGRAM_RANGE, # Use 1-grams + 2-grams. 'dtype': 'int32', 'strip_accents': 'unicode', 'decode_error': 'replace', 'analyzer': TOKEN_MODE, # Split text into word tokens. 'min_df': MIN_DOCUMENT_FREQUENCY, } vectorizer = TfidfVectorizer(**kwargs) # Learn vocabulary from training texts and vectorize training texts. x_train = vectorizer.fit_transform(train_texts) # Vectorize validation texts. x_val = vectorizer.transform(val_texts) # Select top 'k' of the vectorized features. selector = SelectKBest(f_classif, k=min(TOP_K, x_train.shape[1])) selector.fit(x_train, train_labels) x_train = selector.transform(x_train) x_val = selector.transform(x_val) x_train = x_train.astype('float32') x_val = x_val.astype('float32') return x_train, x_val 

Answer: There's a similar open issue that you can find here. Solution proposed is use Tensorflow version 2.1.0 and Keras version 2.3.1.
Question: I'm trying to use this library for setting up a YAML config file for a python project, but I don't want to use the suggested directories for configuration e.g. ~/.config/app or /etc/app for linux. I've tried setting the path using an environment variable as outlined in the documentation here. Does anybody have any experience of getting this to work using the environment variables? I can't really understand why the API doesn't just let you pass a file path, this approach seems unnecessarily complex to me. I suspect there is a good reason I just don't understand! I would have thought in most cases the config file will be in your python project directory?

Answer: I'm experimenting with the library and so far in order to put a config.yaml file in the root folder of my script I just did that: import confuse class MyConfiguration(confuse.Configuration): def config_dir(self): return './' config = MyConfiguration('SplitwiseToBuckets') print(config)  Quite rude I know but for what I want so far it works..! :D
Question: I want to find the mode of the task column in this dataframe: +-----+-----------------------------------------+ | id | task | +-----+-----------------------------------------+ | 101 | [person1, person1, person3] | | 102 | [person1, person2, person3] | | 103 | null | | 104 | [person1, person2] | | 105 | [person1, person1, person2, person2] | | 106 | null | +-----+-----------------------------------------+  If there are multiple modes, I want to display all of them. Can someone please help me get this output: +-----+-----------------------------------------+---------------------------+ | id | task | mode | +-----+-----------------------------------------+---------------------------+ | 101 | [person1, person1, person3] |[person1] | | 102 | [person1, person2, person3] |[person1, person2, person3]| | 103 | null |[] | | 104 | [person1, person2] |[person1, person2] | | 105 | [person1, person1, person2, person2] |[person1, person2] | | 106 | null |[] | +-----+-----------------------------------------+---------------------------+  This is my first question here. Any help or hint is greatly appreciated. Thank you.

Answer: Using Spark 2.3: You can solve this using a custom UDF. For the purposes of getting multiple mode values, I'm using a Counter. I use the except block in the UDF for the null cases in your task column.  (For Python 3.8+ users, there is a statistics.multimode() in-built function you can make use of) Your dataframe: from pyspark.sql.types import * from pyspark.sql import functions as F from pyspark.sql.functions import * schema = StructType([StructField("id", IntegerType()), StructField("task", ArrayType(StringType()))]) data = [[101, ["person1", "person1", "person3"]], [102, ["person1", "person2", "person3"]], [103, None], [104, ["person1", "person2"]], [105, ["person1", "person1", "person2", "person2"]], [106, None]] df = spark.createDataFrame(data,schema=schema)  Operation: from collections import Counter def get_multi_mode_list(input_array): multi_mode = [] counter_var = Counter(input_array) try: temp = counter_var.most_common(1)[0][1] except: temp = counter_var.most_common(1) for i in counter_var: if input_array.count(i) == temp: multi_mode.append(i) return(list(set(multi_mode))) get_multi_mode_list_udf = F.udf(get_multi_mode_list, ArrayType(StringType())) df.withColumn("multi_mode", get_multi_mode_list_udf(col("task"))).show(truncate=False)  Output: +---+------------------------------------+---------------------------+ |id |task |multi_mode | +---+------------------------------------+---------------------------+ |101|[person1, person1, person3] |[person1] | |102|[person1, person2, person3] |[person2, person3, person1]| |103|null |[] | |104|[person1, person2] |[person2, person1] | |105|[person1, person1, person2, person2]|[person2, person1] | |106|null |[] | +---+------------------------------------+---------------------------+
Question: I want to integrate external data into a Django app. Let's say, for example, I want to work with GitHub issues as if they were formulated as normal models within Django. So underneath these objects, I use the GitHub API to retrieve and store data. In particular, I also want to be able to reference the GitHub issues from models but not the other way around. I.e., I don't intend to modify or extend the external data directly. The views would use this abstraction to fetch data, but also to follow the references from "normal objects" to properties of the external data. Simple joins would also be nice to have, but clearly there would be limitations. Are there any examples of how to achieve this in an idiomatic way? Ideally, this would be would also be split in a general part that describes the API in general, and a descriptive part of the classes similar to how normal ORM classes are described.

Answer: The django way in this case would be to write a custom "db" backend. This repo looks abandoned but still can lead you to some ideas.
Question: When writing a Python package, I know how to specify other required Python packages in the setup.py file thanks to the field install_requires from setuptools.setup. However, I do not know how to specify external system dependencies that are NOT Python packages, i.e. a commands such as git or cmake (examples) that my package could call via subprocess.call or subprocess.Popen? Do I have to manually check the availability of the commands in my setup.py file, or is there a fancy way to specify system requirements? Edit: I just want to be able to check if the external tools are available, and if not invite the user to install them (by themself). I do not want to manage the installation of external tools when installing the package. Summary of contributions: it seems that setuptools has no support for this, and it would be safer to do the check at runtime (c.f. comments and answers).

Answer: My recommendation would be to check for the presence of those external dependencies not at install-time but at run-time. Either at the start of each run, or maybe at the first run. It's true that you could add this to your setup.py, but the setup.py is not always executed at install-time: for example if your project is packaged as a wheel then it doesn't even contain the setup.py file at all. And even if you do not distribute your project as a wheel, if I am not mistaken pip tends to build a wheel locally anyway and reuse it for the subsequent installations. So although it would be possible to do such checks as part of the setup script at install time (provided that you can guarantee the presence and execution of setup.py), I would say run-time is a safer bet.
Question: I have a kdeplot but I'm struggling to figure out how to create the legend. import matplotlib.patches as mpatches # see the tutorial for how we use mpatches to generate this figure! # Set 'is_workingday' to a boolean array that is true for all working_days is_workingday = daily_counts["workingday"] == "yes" is_not_workingday = daily_counts['workingday'] == "no" # Bivariate KDEs require two data inputs. # In this case, we will need the daily counts for casual and registered riders on workdays casual_workday = daily_counts.loc[is_workingday, 'casual'] registered_workday = daily_counts.loc[is_workingday, 'registered'] # Use sns.kdeplot on the two variables above to plot the bivariate KDE for weekday rides sns.kdeplot(casual_workday, registered_workday, color = "red", cmap = "Reds", hue = "workingday", legend = True) # Repeat the same steps above but for rows corresponding to non-workingdays casual_non_workday = daily_counts.loc[is_not_workingday, 'casual'] registered_non_workday = daily_counts.loc[is_not_workingday, 'registered'] # Use sns.kdeplot on the two variables above to plot the bivariate KDE for non-workingday rides sns.kdeplot(casual_non_workday, registered_non_workday, color = 'blue', cmap = "Blues", legend = True, shade = False)  Gets me this:  I'm trying to get this: 

Answer: The other answer works nice when one single color is used per kdeplot. In case a colormap such as 'Reds' is used, this would show a very light red. A custom colormap can show a color from the middle of the range: from matplotlib import pyplot as plt import matplotlib.patches as mpatches import seaborn as sns import numpy as np casual_workday = np.random.randn(100) * 1.2 registered_workday = 0.8 * np.random.randn(100) + casual_workday * 0.2 + 1 sns.kdeplot(x=casual_workday, y=registered_workday, color="red", cmap="Reds", shade=False) casual_non_workday = np.random.randn(100) * 1.6 registered_non_workday = 0.5 * np.random.randn(100) + casual_non_workday * 0.5 - 1 sns.kdeplot(x=casual_non_workday, y=registered_non_workday, cmap="Blues", shade=False) handles = [mpatches.Patch(facecolor=plt.cm.Reds(100), label="Workday"), mpatches.Patch(facecolor=plt.cm.Blues(100), label="Non-workday")] plt.legend(handles=handles) plt.show()
Question: Let's say I have an array X of shape (6, 2) like this: import numpy as np X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [9, 10], [11, 12]])  I want to reshape it to an array of shape (3, 2, 2), so I did this: X.reshape(3, 2, 2)  And got: array([[[ 1, 2], [ 3, 4]], [[ 5, 6], [ 7, 8]], [[ 9, 10], [11, 12]]])  However, I need my data in a different format. To be precise, I want to end up wth: array([[[ 1, 2], [ 7, 8]], [[ 3, 4], [ 9, 10]], [[ 5, 6], [11, 12]]])  Should I be using reshape for this or something else? What's the best way to do this in Numpy?

Answer: You have to set the order option: >>> X.reshape(3, 2, 2, order='F') array([[[ 1, 2], [ 7, 8]], [[ 3, 4], [ 9, 10]], [[ 5, 6], [11, 12]]])   ‘F’ means to read / write the elements using Fortran-like index order, with the first index changing fastest, and the last index changing slowest.  see: https://numpy.org/doc/stable/reference/generated/numpy.reshape.html
Question: pandas version: 1.2 I have a dataframe that columns as 'float64' with null values represented as pd.NAN. Is there way to round without converting to string then decimal: df = pd.DataFrame([(.21, .3212), (.01, .61237), (.66123, .03), (.21, .18),(pd.NA, .18)], columns=['dogs', 'cats']) df dogs cats 0 0.21 0.32120 1 0.01 0.61237 2 0.66123 0.03000 3 0.21 0.18000 4 <NA> 0.18000  Here is what I wanted to do, but it is erroring: df['dogs'] = df['dogs'].round(2)   TypeError: float() argument must be a string or a number, not 'NAType'  Here is another way I tried but this silently fails and no conversion occurs: tn.round({'dogs': 1}) dogs cats 0 0.21 0.32120 1 0.01 0.61237 2 0.66123 0.03000 3 0.21 0.18000 4 <NA> 0.18000 

Answer: df['dogs'] = df['dogs'].apply(lambda x: round(x,2) if str(x) != '<NA>' else x)
Question: I have a dataset like this my_data= [['Manchester', '23', '80', 'CM', 'Manchester', '22', '79', 'RM', 'Manchester', '19', '76', 'LB'], ['Benfica', '26', '77', 'CF', 'Benfica', '22', '74', 'CDM', 'Benfica', '17', '70', 'RB'], ['Dortmund', '24', '75', 'CM', 'Dortmund', '18', '74', 'AM', 'Dortmund', '16', '69', 'LM'] ]  I know that using train_test_split from sklearn.cross_validation, and I've tried with this from sklearn.model_selection import train_test_split train, test = train_test_split(my_data, test_size = 0.2)  The result just split into test and train. I wish to divide it to 3 separate sets with randomized data. Expected: Test, Train, Valid

Answer: You can simply use train_test split twice X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1) X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.25, random_state=1)  also, the answer can be found here
Question: What I'm aiming at is - if I ran the following test: def test_func(): with pytest.raises(APIError): func()  and the func() did not raise APIError - I want to get custom message to the output, e.g. "No APIError caught" Pytest had a feature specifically for this: with raises(SomeError, message="Custom message here"): pass  https://docs.pytest.org/en/4.2.1/assert.html#assertions-about-expected-exceptions but for some reason it didn't make it to further versions (current is 6.0.2)

Answer: You can replicate this behavior using the pytest.fail function after the function you expect to raise an exception, since that will only run if an exception is not raised. The deprecation notice explains the reasoning for its removal, and offers this alternate approach: import pytest def func(): return def test_func(): with pytest.raises(ValueError): func() pytest.fail("Oh no!")  $ pytest test.py collected 1 item test.py F [100%] ==================================================================================================== FAILURES ===================================================================================================== ____________________________________________________________________________________________________ test_func ____________________________________________________________________________________________________ def test_func(): with pytest.raises(ValueError): func() > pytest.fail("Oh no!") E Failed: Oh no! test.py:9: Failed ============================================================================================= short test summary info ============================================================================================= FAILED test.py::test_func - Failed: Oh no! ================================================================================================ 1 failed in 0.16s ================================================================================================
Question: I have an array of integers: [int1, int2, ..., intn]  I want to count how many non-zero bits are in the binary representation of these integers. For example: bin(123) -> 0b1111011, there are 6 non-zero bits  Of course I can loop over list of integers, use bin() and count('1') functions, but I'm looking for vectorized way to do it.

Answer: Assuming your array is a, you can simply do: np.unpackbits(a.view('uint8')).sum()  example: a = np.array([123, 44], dtype=np.uint8) #bin(a) is [0b1111011, 0b101100] np.unpackbits(a.view('uint8')).sum() #9   Comparison using benchit: #@Ehsan's solution def m1(a): return np.unpackbits(a.view('uint8')).sum() #@Valdi_Bo's solution def m2(a): return sum([ bin(n).count('1') for n in a ]) in_ = [np.random.randint(100000,size=(n)) for n in [10,100,1000,10000,100000]]  m1 is significantly faster.
Question: I am making a simple flask API for uploading an image and do some progresses then store it in the data base as binary, then i want to download it by using send_file() function but, when i am passing an image like a bytes it gives me an error:  return send_file(BytesIO.read(image.data), attachment_filename='f.jpg', as_attachment=True) TypeError: descriptor  'read' requires a '_io.BytesIO' object but received a 'bytes'  and My code for upload an image as follow: @app.route('/upload', methods=['POST']) def upload(): images = request.files.getlist('uploadImages') n = 0 for image in images: fileName = image.filename.split('.')[0] fileFormat = image.filename.split('.')[1] imageRead = image.read() img = BytesIO(imageRead) with graph.as_default(): caption = generate_caption_from_file(img) newImage = imageDescription(name=fileName, format=fileFormat, description=caption, data=imageRead) db.session.add(newImage) db.session.commit() n = n + 1 return str(n) + ' Image has been saved successfully'  And my code for downloading an image: @app.route('/download/<int:id>') def download(id): image = imageDescription.query.get_or_404(id) return send_file(BytesIO.read(image.data), attachment_filename='f.jpg', as_attachment=True)  any one can help please???

Answer: It seems you are confused io.BytesIO. Let's look at some examples of using BytesIO. >>> from io import BytesIO >>> inp_b = BytesIO(b'Hello World', ) >>> inp_b <_io.BytesIO object at 0x7ff2a71ecb30> >>> inp.read() # read the bytes stream for first time b'Hello World' >>> inp.read() # now it is positioned at the end so doesn't give anything. b'' >>> inp.seek(0) # position it back to begin >>> BytesIO.read(inp) # This is same as above and prints bytes stream b'Hello World' >>> inp.seek(0) >>> inp.read(4) # Just read upto four bytes of stream. >>> b'Hell'  This should give you an idea of how read on BytesIO works. I think what you need to do is this. return send_file( BytesIO(image.data), mimetype='image/jpg', as_attachment=True, attachment_filename='f.jpg' )
Question: I have 12 people who i need to divide into 2 different teams. What i need to do is pick random 6 numbers between 0 and 11 for the first team and do the same for the second one with no overlap. What is the most efficient way to do this? import random A = random.choice([x for x in range(12)]) B = random.choice([x for x in range(12) if x != A]) C = random.choice([x for x in range(12) if (x != A) and (x != B)]) team1 = random.sample(range(0, 12), 6) team2 = random.sample(range(0, 12), 6)  This is what i wrote so far. Any help is appreciated.

Answer: You can use sets and set difference, like this: import random all_players = set(range(12)) team1 = set(random.sample(all_players, 6)) team2 = all_players - team1 print(team1) print(team2)  Example Output: {1, 5, 8, 9, 10, 11} {0, 2, 3, 4, 6, 7}
Question: In a dataframe with around 40+ columns I am trying to change dtype for first 27 columns from float to int by using iloc: df1.iloc[:,0:27]=df1.iloc[:,0:27].astype('int')  However, it's not working. I'm not getting any error, but dtype is not changing as well. It still remains float. Now the strangest part: If I first change dtype for only 1st column (like below): df1.iloc[:,0]=df1.iloc[:,0].astype('int')  and then run the earlier line of code: df1.iloc[:,0:27]=df1.iloc[:,0:27].astype('int')  It works as required. Any help to understand this and solution to same will be grateful. Thanks!

Answer: I guess it is a bug in 1.0.5. I tested on my 1.0.5. I have the same issue as yours. The .loc also has the same issue, so I guess pandas devs break something in iloc/loc. You need to update to latest pandas or use a workaround. If you need a workaround, using assignment as follows df1[df1.columns[0:27]] = df1.iloc[:, 0:27].astype('int')  I tested it. Above way overcomes this bug. It will turn first 27 columns to dtype int32
Question: whenever I run the server or executing any commands in the terminal this error is showing in the terminal. The server is running and the webpage is working fine but when I quit the server or run any commands(like python manage.py migrate) this error is showing.  `Watching for file changes with StatReloader Performing system checks... System check identified no issues (0 silenced). September 21, 2020 - 12:42:24 Django version 3.0, using settings 'djangoblog.settings' Starting development server at http://127.0.0.1:8000/ Quit the server with CTRL-BREAK. Traceback (most recent call last): File "manage.py", line 22, in <module> main() File "manage.py", line 18, in main execute_from_command_line(sys.argv) File "C:\Python37\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line utility.execute() File "C:\Python37\lib\site-packages\django\core\management\__init__.py", line 395, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "C:\Python37\lib\site-packages\django\core\management\base.py", line 341, in run_from_argv connections.close_all() File "C:\Python37\lib\site-packages\django\db\utils.py", line 230, in close_all connection.close() File "C:\Python37\lib\site-packages\django\utils\asyncio.py", line 24, in inner return func(*args, **kwargs) File "C:\Python37\lib\site-packages\django\db\backends\sqlite3\base.py", line 261, in close if not self.is_in_memory_db(): File "C:\Python37\lib\site-packages\django\db\backends\sqlite3\base.py", line 380, in is_in_memory_db return self.creation.is_in_memory_db(self.settings_dict['NAME']) File "C:\Python37\lib\site-packages\django\db\backends\sqlite3\creation.py", line 12, in is_in_memory_db return database_name == ':memory:' or 'mode=memory' in database_name TypeError: argument of type 'WindowsPath' is not iterable ` 

Answer: I got this cleared by changing DATABASES in settings.py file: change 'NAME': BASE_DIR / 'db.sqlite3',  to 'NAME': str(os.path.join(BASE_DIR, "db.sqlite3"))   this works DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': str(os.path.join(BASE_DIR, "db.sqlite3")) } }