eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I'm trying to send a PIL Image object to a discord chat (I don't want to save the file though) I have a function that gathers images from the internet, joins them together vertically and then return a PIL Image object.  The code below creates a file image from the PIL Image object on my local machine and then sends it to a Discord chat. I don't want to constantly be recreating and saving the file image on my machine. How can I just send the PIL Image object instead of having to save the image every time I send a request? from PIL import Image from io import BytesIO import requests import discord # Initializes Discord Client client = discord.Client() # List of market indexes indexes = [ 'https://finviz.com/image.ashx?dow', 'https://finviz.com/image.ashx?nasdaq', 'https://finviz.com/image.ashx?sp500' ] # Returns a vertical image of market indexes def create_image(): im = [] for index in indexes: response = requests.get(index) im.append(Image.open(BytesIO(response.content))) dst = Image.new('RGB', (im[0].width, im[0].height + im[1].height + im[2].height)) dst.paste(im[0], (0, 0)) dst.paste(im[1], (0, im[0].height)) dst.paste(im[2], (0, im[0].height + im[1].height)) return dst # Prints when bot is online @client.event async def on_ready(): print('{0.user} is online'.format(client)) # Uploads vertical image of market indexes when requested @client.event async def on_message(message): if message.content.startswith('^index'): create_image().save('index.png') await message.channel.send(file=discord.File('index.png'))  SOLUTION: @client.event async def on_message(message): if message.content.startswith('^index'): with BytesIO() as image_binary: create_image().save(image_binary, 'PNG') image_binary.seek(0) await message.channel.send(file=discord.File(fp=image_binary, filename='image.png')) 

Answer: Posting my solution as a separate answer. Thanks Ceres for the recommendation. @client.event async def on_message(message): if message.content.startswith('^index'): with BytesIO() as image_binary: create_image().save(image_binary, 'PNG') image_binary.seek(0) await message.channel.send(file=discord.File(fp=image_binary, filename='image.png'))
Question: Is there any way I can recursively get all keys in h5 file using python library h5py? I tried using the code below  import h5py h5_data = h5py.File(h5_file_location, 'r') print(h5_data.keys())  but it only print the top level keys of the h5 file.

Answer: Some of the keys returned by keys() on a Group may be Datasets some may be sub Groups. In order to find all keys you need to recurse the Groups. Here is a simple script to do that: import h5py def allkeys(obj): "Recursively find all keys in an h5py.Group." keys = (obj.name,) if isinstance(obj, h5py.Group): for key, value in obj.items(): if isinstance(value, h5py.Group): keys = keys + allkeys(value) else: keys = keys + (value.name,) return keys h5 = h5py.File('/dev/null', 'w') h5.create_group('g1') h5.create_group('g2') h5.create_dataset('d1', (10,), 'i') h5.create_dataset('d2', (10, 10,), 'f') h5['g1'].create_group('g1') h5['g1'].create_dataset('d1', (10,), 'i') h5['g1'].create_dataset('d2', (10,), 'f') h5['g1/g1'].attrs['a'] = 'b' print(allkeys(h5))  Gives: ('/', '/d1', '/d2', '/g1', '/g1/d1', '/g1/d2', '/g1/g1', '/g2')
Question: I recently discovered plotly and find it really good for graphing, now I have a problem which I want to save multiple plot into a single html, how to do it please? *I want to save multiple plot, i.e fig, fig1, fig 2 and so on, NOT one subplot which has multiple plot in it, because I found that the plot within subplot is too small.

Answer: In the Plotly API there is a function to_html which returns HTML of the figure. Moreover, you can set option param full_html=False which will give you just DIV containing figure. You can just write multiple figures to one HTML by appending DIVs containing figures: with open('p_graph.html', 'a') as f: f.write(fig1.to_html(full_html=False, include_plotlyjs='cdn')) f.write(fig2.to_html(full_html=False, include_plotlyjs='cdn')) f.write(fig3.to_html(full_html=False, include_plotlyjs='cdn'))  https://plot.ly/python-api-reference/generated/plotly.io.to_html.html You can also use Beautiful Soup to do DOM manipulation and insert DIV exactly where you need it in the HTML. https://beautiful-soup-4.readthedocs.io/en/latest/#append
Question: after updating everything in conda, pip can't install anything conda update -n base conda conda update --all  when install or upgrade anything, this error is show  $ pip install --upgrade HDF5 Traceback (most recent call last): File "C:\ProgramData\Anaconda3\Scripts\pip-script.py", line 10, in <module> sys.exit(main()) File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_internal\main.py", line 45, in main command = create_command(cmd_name, isolated=("--isolated" in cmd_args)) File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_internal\commands\__init__.py", line 96, in create_command module = importlib.import_module(module_path) File "C:\ProgramData\Anaconda3\lib\importlib\__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 994, in _gcd_import File "<frozen importlib._bootstrap>", line 971, in _find_and_load File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 665, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_internal\commands\install.py", line 23, in <module> from pip._internal.cli.req_command import RequirementCommand File "C:\ProgramData\Anaconda3\lib\site-packages\pip\_internal\cli\req_command.py", line 17, in <module> from pip._internal.index import PackageFinder ImportError: cannot import name 'PackageFinder'  any help please. thank you.

Answer: It seems that this works. Reinstall the latest version of pip: $ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python get-pip.py  When you’re done, delete the installation script: $ rm get-pip.py
Question: I´m having hard time to use GraphQL with Python since the suggested library: gql is completely undocumented. How ever I found out that to provide the api url I need to pass a RequestsHTTPTransport object to Client like this: client = Client(transport=RequestsHTTPTransport(url='https://some.api.com/v3/graphql'))  but how to provide credentials like the Bearer Key? PS I noticed that it RequestsHTTPTransport accepts also a auth param which is described as: :param auth: Auth tuple or callable to enable Basic/Digest/Custom HTTP Auth  how ever I still can not find out how to create this tuple or callable to work with a Bearer Key :( Thanks in advise

Answer: You can add it in the headers. reqHeaders = { 'x-api-key' : API_KEY, 'Authorization': 'Bearer ' + TOKEN_KEY // This is the key } _transport = RequestsHTTPTransport( url=API_ENDPOINT, headers = reqHeaders, use_json=True, ) client = Client( transport = _transport, fetch_schema_from_transport=True, )
Question: I'm using Django 2.0, Python 3.7, and MySql 5. I recently installed the django_address module. I noticed when I ran my initial migration based on my models.py file ... from django.db import models from address.models import AddressField from phonenumber_field.modelfields import PhoneNumberField class CoopType(models.Model): name = models.CharField(max_length=200, null=False) class Meta: unique_together = ("name",) class Coop(models.Model): type = models.ForeignKey(CoopType, on_delete=None) address = AddressField(on_delete=models.CASCADE) enabled = models.BooleanField(default=True, null=False) phone = PhoneNumberField(null=True) email = models.EmailField(null=True) web_site = models.TextField()  It created some address tables, including ... mysql> show create table address_country; +-----------------+---------------------------------------------------+ | Table | Create Table | +-----------------+---------------------------------------------------+ | address_country | CREATE TABLE `address_country` ( | | | `id` int(11) NOT NULL AUTO_INCREMENT, | | | `name` varchar(40) COLLATE utf8_bin NOT NULL, | | | `code` varchar(2) COLLATE utf8_bin NOT NULL, | | | PRIMARY KEY (`id`), | | | UNIQUE KEY `name` (`name`) | | | ) ENGINE=InnoDB | | | DEFAULT CHARSET=utf8 COLLATE=utf8_bin | +-----------------+---------------------------------------------------+  However, this table has no data in it. Is there a way to obtain seed data for the table generated by the module or do I need to dig it up on my own?

Answer: I'd suggest you write a simple management command that imports data from pycountry into your address models (approach borrowed from here). pycountry is a wrapper around the ISO standard list of countries - i.e., it's about as canonical a list of countries as you're going to get. A management command to populate all the countries into your Country model would look something like this: import pycountry from django.core.management.base import BaseCommand, CommandError from address.models import Country class Command(BaseCommand): help = "Populates address.Country with data from pycountry." def handle(self, *args, **options): countries = [ Country( code=country.alpha_2, name=country.name[:40], # NOTE - concat to 40 chars because of limit on the model field ) for country in pycountry.countries ] Country.objects.bulk_create(countries) self.stdout.write("Successfully added %s countries." % len(countries))  This will populate your model with the ISO list of countries.  One caveat here is that the address.Country.name field is limited to 40 characters (which seems like a questionable design decision to me, along with the decision not to make country codes unique - ISO 2-letter codes definitely are unique), so the script above truncates the name to fit. If this is a problem for you I suggest you set up your own address models, borrowing from django-address and raising the character limit.
Question: Tried running pip3 install torch===1.4.0 torchvision===0.5.0 -f https://download.pytorch.org/whl/torch_stable.html first, taken from PyTorch website which resulted in No matching distribution found for torch===1.4.0  and Could not find a version that satisfies the requirement torch===1.4.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)  Finally downloaded the .whl file from downloads page and tried installing locally like so 'C:\Users\Raf\AppData\Local\Programs\Python\Python38\Scripts\pip.exe' install torch-1.4.0+cpu-cp37-cp37m-win_amd64.whl after which I get torch-1.4.0+cpu-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform. using 64 Python 3.8, on 64 bit Windows

Answer:  using 64 Python 3.8  but you downloaded the cp37 whl which is for python 3.7. There is currently no whl file available for python 3.8. So either install from source (probably not recommended), install a different python version or create a virtual environment with python 3.7 Update There is now: https://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp38-cp38-win_amd64.whl
Question: I have a large matrix (~200 million rows) describing a list of actions that occurred every day (there are ~10000 possible actions). My final goal is to create a co-occurrence matrix showing which actions happen during the same days.  Here is an example dataset: data = {'date': ['01', '01', '01', '02','02','03'], 'action': [100, 101, 989855552, 100, 989855552, 777]} df = pd.DataFrame(data, columns = ['date','action'])  I tried to create a sparse matrix with pd.get_dummies, but unravelling the matrix and using groupby on it is extremely slow, taking 6 minutes for just 5000 rows.  # Create a sparse matrix of dummies dum = pd.get_dummies(df['action'], sparse = True) df = df.drop(['action'], axis = 1) df = pd.concat([df, dum], axis = 1) # Use groupby to get a single row for each date, showing whether each action occurred. # The groupby command here is the bottleneck. cols = list(df.columns) del cols[0] df = df.groupby('date')[cols].max() # Create a co-occurrence matrix by using dot-product of sparse matrices cooc = df.T.dot(df)  I've also tried:  getting the dummies in non-sparse format; using groupby for aggregation; going to sparse format before matrix multiplication.  But I fail in step 1, since there is not enough RAM to create such a large matrix. I would greatly appreciate your help. 

Answer: I came up with an answer using only sparse matrices based on this post. The code is fast, taking about 10 seconds for 10 million rows (my previous code took 6 minutes for 5000 rows and was not scalable).  The time and memory savings come from working with sparse matrices until the very last step when it is necessary to unravel the (already small) co-occurrence matrix before export.  ## Get unique values for date and action date_c = CategoricalDtype(sorted(df.date.unique()), ordered=True) action_c = CategoricalDtype(sorted(df.action.unique()), ordered=True) ## Add an auxiliary variable df['count'] = 1 ## Define a sparse matrix row = df.date.astype(date_c).cat.codes col = df.action.astype(action_c).cat.codes sparse_matrix = csr_matrix((df['count'], (row, col)), shape=(date_c.categories.size, action_c.categories.size)) ## Compute dot product with sparse matrix cooc_sparse = sparse_matrix.T.dot(sparse_matrix) ## Unravel co-occurrence matrix into dense shape cooc = pd.DataFrame(cooc_sparse.todense(), index = action_c.categories, columns = action_c.categories)
Question: Although everything works fine, I would like to know whether there is a way to fix what is provoking this warning:  plotly.graph_objs.Line is deprecated. Please replace it with one of the following more specific types  plotly.graph_objs.scatter.Line plotly.graph_objs.layout.shape.Line etc.  

Answer: Fixing the warning regarding deprecated functions may variate, depending on the packages at use. In my particular case, I was using the "Line" function from "plotly" package. This function was being called from another package. In the latter package there was a .py file (I used "IDLE (Python 3.8 64-bit)" to edit it) that had the following code line: from plotly.graph_objs import Line  Then, I first tried replacing that line with the suggestions provided by the warning. After a couple of tries, I ended up using the code line: from plotly.graph_objs.scatter.marker import Line  My final script works fine like in the beginning, but this time with no warnings at all. Note: In my case, the packages were installed in "C:\Users\NIP\AppData\Roaming\Python\Python38\site-packages" I hope this helps.
Question: I'm working on the mushroom classification data set (found here: https://www.kaggle.com/uciml/mushroom-classification). I'm trying to split my data into training and testing sets for my models, however if i use the train_test_split method my models always achieve 100% accuracy. This is not the case when i split my data manually. x = data.copy() y = x['class'] del x['class'] x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33) model = xgb.XGBClassifier() model.fit(x_train, y_train) predictions = model.predict(x_test) print(confusion_matrix(y_test, predictions)) print(accuracy_score(y_test, predictions))  This produces: [[1299 0] [ 0 1382]] 1.0  If I split the data manually I get a more reasonable result. x = data.copy() y = x['class'] del x['class'] x_train = x[0:5443] x_test = x[5444:] y_train = y[0:5443] y_test = y[5444:] model = xgb.XGBClassifier() model.fit(x_train, y_train) predictions = model.predict(x_test) print(confusion_matrix(y_test, predictions)) print(accuracy_score(y_test, predictions))  Result: [[2007 0] [ 336 337]] 0.8746268656716418  What could be causing this behaviour? Edit: As per request I'm including shapes of slices. train_test_split: x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.33) print(x_train.shape) print(y_train.shape) print(x_test.shape) print(y_test.shape)  Result: (5443, 64) (5443,) (2681, 64) (2681,)  Manual split: x_train = x[0:5443] x_test = x[5444:] y_train = y[0:5443] y_test = y[5444:] print(x_train.shape) print(y_train.shape) print(x_test.shape) print(y_test.shape)  Result: (5443, 64) (5443,) (2680, 64) (2680,)  I've tried defining my own split function and the resulting split also results in 100% classifier accuracy. Here's the code for the split def split_data(dataFrame, testRatio): dataCopy = dataFrame.copy() testCount = int(len(dataFrame)*testRatio) dataCopy = dataCopy.sample(frac = 1) y = dataCopy['class'] del dataCopy['class'] return dataCopy[testCount:], dataCopy[0:testCount], y[testCount:], y[0:testCount] 

Answer: You got lucky there on your train_test_split. The split you are doing manually may be having the most unseen data, which is doing better validation than the train_test_split which internally shuffled the data to split it. For better validation use K-fold cross validation, which will allow to verify the model accuracy with each of the different parts in your data as your test and rest part as train.
Question: I am using ORM peewee for sqlite in Python. I would like to create table Item with field parent_id that will be foreign key to the Item: from peewee import * db = SqliteDatabase("data.db") class Item(Model): id = AutoField() parent_id = ForeignKeyField(Item, null = True) class Meta: database = db db.create_tables([Item])  However, there is error because of circular foreign key:  NameError: free variable 'Item' referenced before assignment in enclosing scope   For this case, there is DeferredForeignKey in peewee: from peewee import * db = SqliteDatabase("data.db") class Item(Model): id = AutoField() parent_id = DeferredForeignKey("Item", null = True) class Meta: database = db db.create_tables([Item]) Item._schema.create_foreign_key(Item.parent_id)  Unfortunately, there is no ADD CONSTRAINT in sqlite, so another error appears:  peewee.OperationalError: near "CONSTRAINT": syntax error  Is there any way to create circural foreign key in sqlite using peewee, or I have to use plain integer instead of foreign key or use native SQL instead of ORM?

Answer: This is documented very clearly: http://docs.peewee-orm.com/en/latest/peewee/models.html#self-referential-foreign-keys You just put 'self' as the identifier: class Item(Model): id = AutoField() parent = ForeignKeyField('self', backref='children', null=True) class Meta: database = db  You do not need to mess with any deferred keys or anything.
Question: I am trying to write a simple python script that sends a gcode command to my wanhao D9 motherboard printer, running Marlin. I am running the python script on a raspberry pi that is connected to the printer via USB. import serial ser = serial.Serial("/dev/ttyUSB0", 115200) ser.write("G28\n")  I have read over 20 forum pages with simular problems and tried their answers such as changing the baud rate to 250000 and the following changes to the write function parameter: ser.write("G28\r\n") ser.write(b'G28\r\n') ser.write(b'G28\n') ser.write(b'G28') ser.write("G28")  I have tried all these combinations, and I have also added: time.sleep(5)  along with the relevant import statement for the time module at the top of my file. I added this line of code between my ser declaration and my ser.write function call. I have also tried adding: ser.close()  to see if that would make a difference but it has not, as I know this is best practice anyway. No matter what combination of this code I used, when I run my python script my printer seems to restart (the screen changes from the home page to the opening wanhao logo and back to the home page) I look forward to any help anyone can give me in regards to my code and what I may be doing wrong.

Answer: Adding an extra sleep period after my command fixed my issue. I can also now read back the initial set up feedback from the printer. My final code without this is: import serial import time ser = serial.Serial('/dev/ttyUSB0', 115200) time.sleep(2) ser.write("G28\r\n") time.sleep(1) ser.close()  Thank you, to the users in this post for guiding me in the right direction Full examples of using pySerial package This code works for me, I hope someone else finds it useful in the future too.
Question: How can I insert for loops or if expressions inside an f-string? I thought initially of doing something like this for if expressions: f'{a:{"s" if CONDITION else "??"}}'  What I would like to do though is something like: Example 1 f'{key: value\n for key, value in dict.items()}'  result: if dict = {'a': 1, 'b': 2} a: 1 b: 2  or Example 2 c = 'hello' f'{c} {name if name else "unknown"}'  result: if name exists, e.g. name = 'Mike' hello Mike  otherwise hello unknown  Can this be done and if yes how?

Answer: Both ternaries ("if expressions") and comprehensions ("for expressions") are allowed inside f-strings. However, they must be part of expressions that evaluate to strings. For example, key: value is a dict pair, and f"{key}: {value}" is required to produce a string. >>> dct = {'a': 1, 'b': 2} >>> newline = "\n" # \escapes are not allowed inside f-strings >>> print(f'{newline.join(f"{key}: {value}" for key, value in dct.items())}') a: 1 b: 2  Note that if the entire f-string is a single format expression, it is simpler to just evaluate the expression directly. >>> print("\n".join(f"{key}: {value}" for key, value in dct.items()))) a: 1 b: 2  Expressions inside format strings still follow their regular semantics. For example, a ternary may test whether an existing name is true. It will fail if the name is not defined. >>> c, name = "Hello", "" >>> f'{c} {name if name else "unknown"}' 'Hello unknown' >>> del name >>> f'{c} {name if name else "unknown"}' NameError: name 'name' is not defined
Question: I am new to python and I am trying to install new packages in Anaconda. I am using anaconda prompt and Windows 10.  Can you please explain what is the difference between conda install with -c anaconda and without it? For example conda install -c anaconda mysqlclient and conda install mysqlclient. Which is better to use when and why? 

Answer: When you use the -c option, you are specifying the channel from which to get the package. The default is -c anaconda, so they are similar. To use packages built locally, you would use -c local. Here is a link for more info: Docs explaining usage of conda install