eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I live in Chittagong, Bangladesh and my time zone is GMT+6. How can i change to this time zone in Django settings?

Answer: You can specify the timezone as 'Asia/Dhaka' in the TIME_ZONE setting [Django-doc]: # settings.py TIME_ZONE = 'Asia/Dhaka' # … Note that if USE_TZ setting [Django-doc] is set to True, then:  When USE_TZ is True, this is the default time zone that Django will use to display datetimes in templates and to interpret datetimes entered in forms.
Question: I am using pytrends library to extract google trends and i am getting the following error:  Addition/subtraction of integers and integer-arrays with Timestamp is no longer supported. Instead of adding/subtracting n, use n * obj.freq  timeframes = [] datelist = pd.date_range('2004-01-01', '2018-01-01', freq="AS") date = datelist[0] while date <= datelist[len(datelist)-1]: start_date = date.strftime("%Y-%m-%d") end_date = (date+4).strftime("%Y-%m-%d") timeframes.append(start_date+' '+end_date) date = date+3 

Answer: You can't sum a date and a number like date+4 because who knows which unit this is, 4h, 4d,... ?  You may use datetime.timedelta, here's an example if you meant days from datetime import timedelta end_date = (date + timedelta(days=4)).strftime("%Y-%m-%d") # ... date = date + timedelta(days=3)
Question: I'm new in python threading and I'm experimenting this: When I run something in threads (whenever I print outputs), it never seems to be running in parallel. Also, my functions take the same time that before using the library concurrent.futures (ThreadPoolExecutor). I have to calculate the gains of some attributes over a dataset (I cannot use libraries). Since I have about 1024 attributes and the function was taking about a minute to execute (and I have to use it in a for iteration) I dicided to split the array of attributes into 10 (just as an example) and run the separete function gain(attribute) separetly for each sub array. So I did the following (avoiding some extra unnecessary code): def calculate_gains(self): splited_attributes = np.array_split(self.attributes, 10) result = {} for atts in splited_attributes: with concurrent.futures.ThreadPoolExecutor() as executor: future = executor.submit(self.calculate_gains_helper, atts) return_value = future.result() self.gains = {**self.gains, **return_value}  Here's the calculate_gains_helper: def calculate_gains_helper(self, attributes): inter_result = {} for attribute in attributes: inter_result[attribute] = self.gain(attribute) return inter_result  Am I doing something wrong? I read some other older posts but I couldn't get any info. Thanks a lot for any help!

Answer: I had this same trouble and fixed by moving the iteration to within the context of the ThreadPoolExecutor, or else, you'll have to wait for the context to finish and start another one. Here is a probably fix for your code: def calculate_gains(self): splited_attributes = np.array_split(self.attributes, 10) result = {} with concurrent.futures.ThreadPoolExecutor() as executor: for atts in splited_attributes: future = executor.submit(self.calculate_gains_helper, atts) return_value = future.result() self.gains = {**self.gains, **return_value}  To demonstrate better what I mean here is a sample code: Below is a non working code. Threads will execute synchronoulsly... from concurrent.futures import ThreadPoolExecutor, as_completed from time import sleep def t(reference): i = 0 for i in range(10): print(f"{reference} :" + str(i)) i+=1 sleep(1) futures = [] refs = ["a", "b", "c"] for i in refs: with ThreadPoolExecutor(max_workers=3) as executor: futures.append(executor.submit(t, i)) for future in as_completed(futures): print(future.result())  Here is the fixed code: from concurrent.futures import ThreadPoolExecutor, as_completed from time import sleep def t(reference): i = 0 for i in range(10): print(f"{reference} :" + str(i)) i+=1 sleep(1) futures = [] refs = ["a", "b", "c"] with ThreadPoolExecutor(max_workers=3) as executor: #swapped for i in refs: #swapped futures.append(executor.submit(t, i)) for future in as_completed(futures): print(future.result())  You can try this on your terminal and check out the outputs.
Question: I'm using FastAPI and currently I return a csv which I read from SQL server with pandas. (pd.read_sql()) However the csv is quite big for the browser and I want to return it with a File response: https://fastapi.tiangolo.com/advanced/custom-response/ (end of the page). I cannot seem to do this without first writing it to a csv file which seems slow and will clutter the filesystem with csv's on every request.  So my questions way, is there way to return a FileResponse from a sql database or pandas dataframe. And if not, is there a way to delete the generated csv files, after it has all been read by the client? Thanks for your help! Kind regards, Stephan

Answer: Based HEAVILY off this https://github.com/tiangolo/fastapi/issues/1277  Turn your dataframe into a stream use a streaming response Modify headers so it's a download (optional)  from fastapi import FastAPI from fastapi.responses import StreamingResponse import io import pandas as pd app = FastAPI() @app.get("/get_csv") async def get_csv(): df = pd.DataFrame(dict(col1 = 1, col2 = 2), index=[0]) stream = io.StringIO() df.to_csv(stream, index = False) response = StreamingResponse(iter([stream.getvalue()]), media_type="text/csv" ) response.headers["Content-Disposition"] = "attachment; filename=export.csv" return response
Question: We use static type checking extensively, but we also need some simple runtime type checking. I'd love to use our static types for that runtime type checking. I've seen typeguard and the other libraries, but I'd prefer to have something simpler. I've tried below, but assert value in expected_type doesn't make sense. How do I create a simple function that will check if a string is in a Python string literal? from typing_extensions import Literal def check_str_in_literal(value: str, expected_type: Literal): assert value in expected_type Gender = Literal["Male", "Female", "Other"] def print_gender(gender: Gender): print(gender) # Unknown string as it's been retrieved from elsewhere strRetrievedFromDB = "Male" # type: ignore check_str_in_literal(strRetrievedFromDB, Gender) print_gender(strRetrievedFromDB) 

Answer: Python 3.8 introduced typing.get_args(tp), making this possible: assert value in get_args(expected_type)
Question: I had a look at this tutorial in the PyTorch docs for understanding Transfer Learning. There was one line that I failed to understand.  After the loss is calculated using loss = criterion(outputs, labels), the running loss is calculated using running_loss += loss.item() * inputs.size(0) and finally, the epoch loss is calculated using running_loss / dataset_sizes[phase]. Isn't loss.item() supposed to be for an entire mini-batch (please correct me if I am wrong). i.e, if the batch_size is 4, loss.item() would give the loss for the entire set of 4 images. If this is true, why is loss.item() being multiplied with inputs.size(0) while calculating running_loss? Isn't this step like an extra multiplication in this case? Any help would be appreciated. Thanks!

Answer: It's because the loss given by CrossEntropy or other loss functions is divided by the number of elements i.e. the reduction parameter is mean by default. torch.nn.CrossEntropyLoss(weight=None, size_average=None, ignore_index=-100, reduce=None, reduction='mean')  Hence, loss.item() contains the loss of entire mini-batch, but divided by the batch size. That's why loss.item() is multiplied with batch size, given by inputs.size(0), while calculating running_loss.
Question: There are many ways to save a model and its weights. It is confusing when there are so many ways and not any source where we can read and compare their properties.  Some of the formats I know are: 1. YAML File - Structure only 2. JSON File - Structure only 3. H5 Complete Model - Keras 4. H5 Weights only - Keras 5. ProtoBuf - Deployment using TensorFlow serving 6. Pickle - Scikit-learn 7. Joblib - Scikit-learn - replacement for Pickle, for objects containing large data.  Discussion: Unlike scikit-learn, Keras does not recommend you save models using pickle. Instead, models are saved as an HDF5 file. The HDF5 file contains everything you need to not only load the model to make predictions (i.e., architecture and trained parameters) but also to restart training (i.e., loss and optimizer settings and the current state). What are other formats to save the model for Scikit-learn, Keras, Tensorflow, and Mxnet? Also what info I am missing about each of the above-discussed formats?

Answer: There are also formats like onnx which basically supports most of the frameworks and helps in removing the confusion of using different formats for different frameworks.
Question: Have tensor like :x.shape = [3, 2, 2]. import torch x = torch.tensor([ [[-0.3000, -0.2926],[-0.2705, -0.2632]], [[-0.1821, -0.1747],[-0.1526, -0.1453]], [[-0.0642, -0.0568],[-0.0347, -0.0274]] ])  I need to take .max() over the 2nd and 3rd dimensions. I expect some like this [-0.2632, -0.1453, -0.0274] as output. I tried to use: x.max(dim=(1,2)), but this causes an error.

Answer: Now, you can do this. The PR was merged (Aug 28 2020) and it is now available in the nightly release. Simply use torch.amax(): import torch x = torch.tensor([ [[-0.3000, -0.2926],[-0.2705, -0.2632]], [[-0.1821, -0.1747],[-0.1526, -0.1453]], [[-0.0642, -0.0568],[-0.0347, -0.0274]] ]) print(torch.amax(x, dim=(1, 2))) # Output: # >>> tensor([-0.2632, -0.1453, -0.0274])   Original Answer As of today (April 11, 2020), there is no way to do .min() or .max() over multiple dimensions in PyTorch. There is an open issue about it that you can follow and see if it ever gets implemented. A workaround in your case would be: import torch x = torch.tensor([ [[-0.3000, -0.2926],[-0.2705, -0.2632]], [[-0.1821, -0.1747],[-0.1526, -0.1453]], [[-0.0642, -0.0568],[-0.0347, -0.0274]] ]) print(x.view(x.size(0), -1).max(dim=-1)) # output: # >>> values=tensor([-0.2632, -0.1453, -0.0274]), # >>> indices=tensor([3, 3, 3]))  So, if you need only the values: x.view(x.size(0), -1).max(dim=-1).values. If x is not a contiguous tensor, then .view() will fail. In this case, you should use .reshape() instead.  Update August 26, 2020 This feature is being implemented in PR#43092 and the functions will be called amin and amax. They will return only the values. This is probably being merged soon, so you might be able to access these functions on the nightly build by the time you're reading this :) Have fun.
Question: I am trying to use plotly (version 4.6.0) to create plots, but having trouble with the markers/size attribute. I am using the Boston housing price dataset in my example. I want to use the value in one of the columns of my dataframe to set a variable size for the marker, but I get an error when I use a direct reference to the column (size='TAX'). I can set the size to a constant (size=1) without issues. I found some examples online, but they generate a ValueError when I try to use them. How can I avoid this error? Code and error are shown below.  import chart_studio.plotly as py import plotly.graph_objs as go from plotly.offline import iplot, init_notebook_mode import cufflinks cufflinks.go_offline(connected=True) init_notebook_mode(connected=True) import pandas as pd from sklearn.datasets import load_boston boston = load_boston() df = pd.DataFrame(boston.data, columns=boston.feature_names) y = boston.target df['RAD_CAT']=df['RAD'].astype(str) df.iplot( x='CRIM', y='INDUS', size='TAX', #size=1, text='RAD', mode='markers', layout=dict( xaxis=dict(type='log', title='CRIM'), yaxis=dict(title='INDUS'), title='CRIM vs INDUS Sized by RAD')) ValueError: Invalid value of type 'builtins.str' received for the 'size' property of scatter.marker Received value: 'TAX' The 'size' property is a number and may be specified as: - An int or float in the interval [0, inf] - A tuple, list, or one-dimensional numpy array of the above 

Answer: import chart_studio.plotly as py import plotly.graph_objs as go from plotly.offline import iplot, init_notebook_mode import cufflinks cufflinks.go_offline(connected=True) init_notebook_mode(connected=True) import pandas as pd from sklearn.datasets import load_boston boston = load_boston() df = pd.DataFrame(boston.data, columns=boston.feature_names) df.iplot( x='CRIM', y='INDUS', size=df['TAX']/20, text='RAD', mode='markers', layout=dict( xaxis=dict(type='log', title='CRIM'), yaxis=dict(title='INDUS'), title='CRIM vs INDUS Sized by TAX'))
Question: The sample data is as follows: unique_list = ['home0', 'page_a0', 'page_b0', 'page_a1', 'page_b1', 'page_c1', 'page_b2', 'page_a2', 'page_c2', 'page_c3'] sources = [0, 0, 1, 2, 2, 3, 3, 4, 4, 7, 6] targets = [3, 4, 4, 3, 5, 6, 8, 7, 8, 9, 9] values = [2, 1, 1, 1, 1, 2, 1, 1, 1, 1, 2]  Using the sample code from the documentation fig = go.Figure(data=[go.Sankey( node = dict( pad = 15, thickness = 20, line = dict(color = "black", width = 0.5), label = unique_list, color = "blue" ), link = dict( source = sources, target = targets, value = values ))]) fig.show()  This outputs the following sankey diagram  However, I would like to get all the values which end in the same number in the same vertical column, just like how the leftmost column has all of it's nodes ending with a 0. I see in the docs that it is possible to move the node positions, however I was wondering if there was a cleaner way to do it other than manually inputting x and y values. Any help appreciated.

Answer: In go.Sankey() set arrangement='snap' and adjust x and y positions in x=<list> and y=<list>. The following setup will place your nodes as requested. Plot:  Please note that the y-values are not explicitly set in this example. As soon as there are more than one node for a common x-value, the y-values will be adjusted automatically for all nodes to be displayed in the same vertical position. If you do want to set all positions explicitly, just set arrangement='fixed' Edit: I've added a custom function nodify() that assigns identical x-positions to label names that have a common ending such as '0' in ['home0', 'page_a0', 'page_b0']. Now, if you as an example change page_c1 to page_c2 you'll get this:  Complete code: import plotly.graph_objects as go unique_list = ['home0', 'page_a0', 'page_b0', 'page_a1', 'page_b1', 'page_c1', 'page_b2', 'page_a2', 'page_c2', 'page_c3'] sources = [0, 0, 1, 2, 2, 3, 3, 4, 4, 7, 6] targets = [3, 4, 4, 3, 5, 6, 8, 7, 8, 9, 9] values = [2, 1, 1, 1, 1, 2, 1, 1, 1, 1, 2] def nodify(node_names): node_names = unique_list # uniqe name endings ends = sorted(list(set([e[-1] for e in node_names]))) # intervals steps = 1/len(ends) # x-values for each unique name ending # for input as node position nodes_x = {} xVal = 0 for e in ends: nodes_x[str(e)] = xVal xVal += steps # x and y values in list form x_values = [nodes_x[n[-1]] for n in node_names] y_values = [0.1]*len(x_values) return x_values, y_values nodified = nodify(node_names=unique_list) # plotly setup fig = go.Figure(data=[go.Sankey( arrangement='snap', node = dict( pad = 15, thickness = 20, line = dict(color = "black", width = 0.5), label = unique_list, color = "blue", x=nodified[0], y=nodified[1] ), link = dict( source = sources, target = targets, value = values ))]) fig.show()
Question: Suppose I have created a model, and my target variable is either 0, 1 or 2. It seems that if I use predict, the answer is either of 0, or 1 or 2. But if I use predict_proba, I get a row with 3 cols for each row as follows, for example  model = ... Classifier # It could be any classifier m1 = model.predict(mytest) m2= model.predict_proba(mytest) # Now suppose m1[3] = [0.6, 0.2, 0.2]  Suppose I use both predict and predict_proba. If in index 3, I get the above result with the result of predict_proba, in index 3 of the result of predict I should see 0. Is this the case? I am trying to understand how using both predict and predict_proba on the same model relate to each other.

Answer:  predict() is used to predict the actual class (in your case one of 0, 1, or 2). predict_proba() is used to predict the class probabilities   From the example output that you shared,  predict() would output class 0 since the class probability for 0 is 0.6. [0.6, 0.2, 0.2] is the output of predict_proba that simply denotes that the class probability for classes 0, 1, and 2 are 0.6, 0.2, and 0.2 respectively.   Now as the documentation mentions for predict_proba, the resulting array is ordered based on the labels you've been using:  The returned estimates for all classes are ordered by the label of classes.  Therefore, in your case where your class labels are [0, 1, 2], the corresponding output of predict_proba will contain the corresponding probabilities. 0.6 is the probability of the instance to be classified as 0 and 0.2 are the probabilities that the instance is categorised as 1 and 2 respectively.  For a more comprehensive explanation, refer to the article What is the difference between predict() and predict_proba() in scikit-learn on TDS.
Question: So I posted this error on a Facebook group, they said I should get pip. I installed pip, when I am wanting to install tkinter it's giving me error: I used this command first : sudo pip install tkinter . . . error: ERROR: Could not find a version that satisfies the requirement tkinter (from versions: none) ERROR: No matching distribution found for tkinter 

Answer: After a day of headache, this worked for me: $ brew install python-tk
Question: i would like to display a pandas dataframe in a interactive way using ipywidgets. So far the code gets some selections and then does some calculation. For this exmaple case, its not really using the input labels. However, my problem is when I would like to display the pandas dataframe, it's not treated as widget. But how can I nicely display then pandas dataframe using widgets? At the end I would like to have a nice table in the main_box here is a code exmaple, which works in any jupyter notebook import pandas as pd import ipywidgets as widgets def button_run_on_click(_): status_label.value = "running...." df = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]]) status_label.value = "" result_box = setup_ui(df) main_box.children = [selection, button_run, status_label, result_box] def setup_ui(df): return widgets.VBox([df]) selection_box = widgets.Box() selection_toggles = [] selected_labels = {} default_labels = ['test1', "test2"] labels = {"test1": "test1", "test2": "test2", "test3": "test3"} def update_selection(change): owner = change['owner'] name = owner.description if change['new']: owner.icon = 'check' selected_labels[name] = labels[name] else: owner.icon = "" selected_labels.pop(name) for k in sorted(labels): o = widgets.ToggleButton(description=k) o.observe(update_selection, 'value') o.value = k in default_labels selection_toggles.append(o) selection_box.children = selection_toggles status_label = widgets.Label() status_label.layout.width = '300px' button_run = widgets.Button(description="Run") main_box = widgets.VBox([selection_box, button_run, status_label]) button_run.on_click(button_run_on_click) display(main_box) 

Answer: from IPython.display import display import ipywidgets as widgets def setup_ui(df): out = widgets.Output() with out: display(df) return out  If you change your setup_ui function to this, you can return an Output widget with your dataframe. BUT, in your button_run_on_click function it appears selection is not defined. Should this be something else?
Question: I have a postgres database that I'm trying to reflect that uses the now standard "Identity" column for primary keys. Here's my table definition: create table class_label ( class_label_id integer PRIMARY KEY GENERATED ALWAYS AS IDENTITY, class_name varchar not null, default_color varchar, created_dttm timestamp default current_timestamp NOT NULL, created_by varchar DEFAULT USER NOT NULL, updated_dttm timestamp default current_timestamp NOT NULL, updated_by varchar DEFAULT user NOT NULL );  And here's my code: from sqlalchemy import create_engine, MetaData, insert, Table, or_, and_, func from sqlalchemy.ext.automap import automap_base from sqlalchemy.orm import scoped_session, sessionmaker import os from sqlalchemy.schema import CreateColumn from sqlalchemy.ext.compiler import compiles @compiles(CreateColumn, 'postgresql') def use_identity(element, compiler, **kw): text = compiler.visit_create_column(element, **kw) text = text.replace("SERIAL", "INT GENERATED BY DEFAULT AS IDENTITY") return text usr = os.environ.get("POSTGRES_USR") pwd = os.environ.get("POSTGRES_PWD") host = os.environ.get("POSTGRES_HOST") engine = create_engine('postgresql://' + usr + ':' + pwd + host, convert_unicode=True) session = scoped_session(sessionmaker(bind=engine)) metadata = MetaData(bind=engine) metadata.reflect(engine, only=['class_label']) Base = automap_base(metadata=metadata) Base.prepare() Class_Label = Base.classes.class_label session.add(Class_Label(class_name="Testing", default_color="red")) session.commit()  When I run my code, I get this error:  sqlalchemy.orm.exc.FlushError: Instance <class_label at 0x1091c4ba8> has a NULL identity key. If this is an auto-generated value, check that the database table allows generation of new primary key values, and that the mapped Column object is configured to expect these generated values. Ensure also that this flush() is not occurring at an inappropriate time, such as within a load() event.  Per https://docs.sqlalchemy.org/en/13/dialects/postgresql.html#postgresql-10-identity-columns I understand that this is somewhat a shortcoming of SQLAlchemy, but I'm wondering if the work-around they suggest can work for auto-mapped/reflected databases and how I would implement it. I'm using SQLAlchemy 1.3.16 and Postgres 11.

Answer: A fix was added in SQLAlchemy 1.4
Question: B = nx.Graph() B.add_nodes_from(data['movie'].unique(), bipartite=0, label='movie') B.add_nodes_from(data['actor'].unique(), bipartite=1, label='actor') B.add_edges_from(edges, label='acted') A = list(nx.connected_component_subgraphs(B))[0]  I am getting the below given error when am trying to use nx.connected_component_subgraphs(G). In the dataset there are two coumns(movie and actor), and it's in the form bipartite graph. I want to get connected components for the movie nodes. --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-16-efff4e6fafc4> in <module> ----> 1 A = list(nx.connected_component_subgraphs(B))[0] AttributeError: module 'networkx' has no attribute 'connected_component_subgraphs' 

Answer: This was deprecated with version 2.1, and finally removed with version 2.4. See these instructions  Use (G.subgraph(c) for c in connected_components(G)) Or (G.subgraph(c).copy() for c in connected_components(G))
Question: I have recently started using PEP 484 and PEP 586 to make my code clearer and more accessible. So far everything was ok, but when I wanted to use Literal from the package typing it appears it couldn't be imported. What is the most surprising is that PyCharm isn't complaining at all for importing it or using it. The code I want to use in the end is looking like that : SomeVar = TypeVar("SomeVar", Literal['choice1'], Literal['choice2'], someType)  It would be used in the cases where you can have a string to describe what you want or an already made solution e.g : def someFunc(my_var: SomeVar = 'choice1'): result = [] if my_var == 'choice1': result.append(...) else: result = my_var return result  I use an Anaconda environment with Python 3.7.7.

Answer: Using Literal in Python 3.8 and later from typing import Literal  Using Literal in all Python versions (1) Literal was added to typing.py in 3.8, but you can use Literal in older versions anyway. First install typing_extensions (pip install typing_extensions) and then from typing_extensions import Literal  This approach is supposed to work also in Python 3.8 and later. Using Literal in all Python versions (2) For completeness, I'm also adding the try-except approach to import Literal: try: from typing import Literal except ImportError: from typing_extensions import Literal  This should also work for all Python versions, given that typing_extensions is installed if you're using Python 3.7 or older.