eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have a Model (which I'm using as an abstract base class), that has some common methods and properties. SQLAlchemy allows creating properties and methods with @hybrid_property and @hybrid_method, but also the standard @property, @classmethod, @staticmethod decorators give me the results I'm after. Are there any advantages and disadvantages using the SQLA decorators over the standard python decorators? When should I use, or shouldn't use, the SQLA decorators?

Answer:  The hybrid provides for an expression that works at both the Python level as well as at the SQL expression level  Let's look at an example: class User(Base): __tablename__ = 'user' id = Column(Integer, primary_key=True) firstname = Column(String(50)) lastname = Column(String(50)) @hybrid_property def fullname(self): return self.firstname + ' ' + self.lastname @property def long_name(self): return self.firstname + ' ' + self.lastname session.add(User(firstname='Brendan', lastname='Simon')) session.commit() # error # print(session.query(User).filter(User.long_name == 'Brendan Simon').first().id) # works fine because @hybrid_property print(session.query(User).filter(User.fullname == 'Brendan Simon').first().id)  Also you can customize SQL expression using @fullname.expression. When should I use, or shouldn't use, the SQLA decorators? I think you will know when you need it. For example, you can use it for fast aliases: class MetaData(Base): __tablename__ = 'meta_data' system_field = Column(String(20)) # a lot of calculations and processing in different places # a lot of fields ...  One day in a few parts system_field was(or will be) renamed to new_field(doesn't matter why, who and when - just fact). You can do something like this as a quick solution: @hybrid_property def new_field(self): return self.system_field @new_field.setter def new_field(self, value: str): self.system_field = value # data from somewhere... # data = {'new_field': 'default', other fields...} # will works fine + in other places will work as `system_field` with old sql queries process_meta(MetaData(**{data}))  So this is really good feature, but if you are thinking about whether you need it or not, then you definitely don't need it.
Question:  RuntimeWarning: greenlet.greenlet size changed, may indicate binary incompatibility. Expected 144 from C header, got 152 from PyObject  And all the workers are being booted.  2020-09-28T14:09:41.864089908Z [2020-09-28 14:09:41 +0000] [31] [INFO] Booting worker with pid: 31 2020-09-28T14:09:43.933141974Z [2020-09-28 14:09:43 +0000] [32] [INFO] Booting worker with pid: 32 2020-09-28T14:09:44.317436676Z [2020-09-28 14:09:44 +0000] [33] [INFO] Booting worker with pid: 33 2020-09-28T14:09:44.795236476Z [2020-09-28 14:09:44 +0000] [34] [INFO] Booting worker with pid: 34  It was working fine a week back or so and now I'm starting to have the problem.

Answer: as https://discuss.redash.io/t/binary-compatibility-issue-with-greenlet/7237 indicates a workaround is greenlet==0.4.16 or upgrade your gevent to 20.9.0 following fix is suggested on the greenlet github page https://github.com/python-greenlet/greenlet/issues/178#issuecomment-697342964 also see following issues  https://github.com/python-greenlet/greenlet/issues/180 https://github.com/python-greenlet/greenlet/issues/182 https://github.com/python-greenlet/greenlet/issues/178
Question: I'd like to use the django.db.models.Q object in a way that the query term is coming from a variable. What i'd like to achieve is identical to this: q = Q(some_field__icontains='sth') Obj.objects.filter(q)  , but the some_field value should come from a variable: field_name='some_field' q = Q('%s__icontains=sth' % field_name) Obj.objects.filter(q)  , but this solution does not give me the correct result of course. I also tried to use dictionary this way: dt = {'%s__icontains' % field_name: 'sth'} q = Q(**dt) Obj.objects.filter(q)  , but this also fails on the result. How could I use the Q object using variables as query term? Thanks.

Answer: You can pass a 2-tuple to a Q object with the name of the fieldname(s) and lookups as first item, and the value as second item: Obj.objects.filter(Q(('%s__icontains' % field_name, 'sth'))) this is probably the most convenient way. That being said the dictionary unpacking, although less elegant, should also work.
Question: I wanted to know how I can change a single node name of a node of a digraph. I am new to networkx and could only find answers on how to change all node names. In my case I am iterating over a graph A to create graph B. p and c are nodes of graph A. The edge (p,c) of graph A contains data I want to add to the node p of B. However, when I am adding the edge data from graph A to the already existing node p of graph B, I would like to update the name of p to be equal to the name of c so I am able to reference it again for the next edge of graph A because it then is the edge (c,x) and I can use the c to reference it again... The relevant part of my code looks like this new_stages = A.in_edge(c, data='stages') stages = B.node[p]['stages'] stages.append(new_stages) <<Update node p to have name of c??>> B.add_node(p, stages=new_stage_set)  Any help is appreciated, thanks!

Answer: You have nx.relabel_nodes for this. Here's a simple use case: G = nx.from_edgelist([('a','b'), ('f','g')]) mapping = {'b':'c'} G = nx.relabel_nodes(G, mapping) G.edges() # EdgeView([('a', 'c'), ('f', 'g')])
Question: Is there a numpy function that splits an array into equal chunks of size m (excluding any remainder which would have a size less than m). I have looked at the function np.array_split but that doesn't let you split by specifying the sizes of the chunks. An example of what I'm looking for is below: X = np.arange(10) split (X, size = 3) -> [ [0,1,2],[3,4,5],[6,7,8], [9] ] split (X, size = 4) -> [ [0,1,2,3],[4,5,6,7],[8,9]] split (X, size = 5) -> [ [0,1,2,3,4],[5,6,7,8,9]] 

Answer: Here's one way with np.split + np.arange/range - def split_given_size(a, size): return np.split(a, np.arange(size,len(a),size))  Sample runs - In [140]: X Out[140]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [141]: split_given_size(X,3) Out[141]: [array([0, 1, 2]), array([3, 4, 5]), array([6, 7, 8]), array([9])] In [143]: split_given_size(X,4) Out[143]: [array([0, 1, 2, 3]), array([4, 5, 6, 7]), array([8, 9])] In [144]: split_given_size(X,5) Out[144]: [array([0, 1, 2, 3, 4]), array([5, 6, 7, 8, 9])]
Question: How can I please convert a multipolygon geometry into a list? I tried this: mycoords=geom.exterior.coords mycoordslist = list(mycoords)  But I get the error:  AttributeError: 'MultiPolygon' object has no attribute 'exterior' 

Answer: You will have to loop over geometries within your MultiPolygon. mycoordslist = [list(x.exterior.coords) for x in geom.geoms]  Note that the result is a list of coords lists.
Question: I have Yolo format bounding box annotations of objects saved in a .txt files. Now I want to load those coordinates and draw it on the image using OpenCV, but I don’t know how to convert those float values into OpenCV format coordinates values I tried this post but it didn’t help, below is a sample example of what I am trying to do Code and output import matplotlib.pyplot as plt import cv2 img = cv2.imread(<image_path>) dh, dw, _ = img.shape fl = open(<label_path>, 'r') data = fl.readlines() fl.close() for dt in data: _, x, y, w, h = dt.split(' ') nx = int(float(x)*dw) ny = int(float(y)*dh) nw = int(float(w)*dw) nh = int(float(h)*dh) cv2.rectangle(img, (nx,ny), (nx+nw,ny+nh), (0,0,255), 1) plt.imshow(img)   Actual Annotations and Image 0 0.286972 0.647157 0.404930 0.371237 0 0.681338 0.366221 0.454225 0.418060  

Answer: There's another Q&A on this topic, and there's this1 interesting comment below the accepted answer. The bottom line is, that the YOLO coordinates have a different centering w.r.t. to the image. Unfortunately, the commentator didn't provide the Python port, so I did that here: import cv2 import matplotlib.pyplot as plt img = cv2.imread(<image_path>) dh, dw, _ = img.shape fl = open(<label_path>, 'r') data = fl.readlines() fl.close() for dt in data: # Split string to float _, x, y, w, h = map(float, dt.split(' ')) # Taken from https://github.com/pjreddie/darknet/blob/810d7f797bdb2f021dbe65d2524c2ff6b8ab5c8b/src/image.c#L283-L291 # via https://stackoverflow.com/questions/44544471/how-to-get-the-coordinates-of-the-bounding-box-in-yolo-object-detection#comment102178409_44592380 l = int((x - w / 2) * dw) r = int((x + w / 2) * dw) t = int((y - h / 2) * dh) b = int((y + h / 2) * dh) if l < 0: l = 0 if r > dw - 1: r = dw - 1 if t < 0: t = 0 if b > dh - 1: b = dh - 1 cv2.rectangle(img, (l, t), (r, b), (0, 0, 255), 1) plt.imshow(img) plt.show()  So, for some Lenna image, that'd be the output, which I think shows the correct coordinates w.r.t. your image:  ---------------------------------------- System information ---------------------------------------- Platform: Windows-10-10.0.16299-SP0 Python: 3.8.5 Matplotlib: 3.3.2 OpenCV: 4.4.0 ----------------------------------------   1Please upvote the linked answers and comments.
Question: what is the difference between using softmax as a sequential layer in tf.keras and softmax as an activation function for a dense layer? tf.keras.layers.Dense(10, activation=tf.nn.softmax)  and tf.keras.layers.Softmax(10) 

Answer: they are the same, you can test it on your own # generate data x = np.random.uniform(0,1, (5,20)).astype('float32') # 1st option X = Dense(10, activation=tf.nn.softmax) A = X(x) # 2nd option w,b = X.get_weights() B = Softmax()(tf.matmul(x,w) + b) tf.reduce_all(A == B) # <tf.Tensor: shape=(), dtype=bool, numpy=True>  Pay attention also when using tf.keras.layers.Softmax, it doesn't require to specify the units, it's a simple activation by default, the softmax is computed on the -1 axis, you can change this if you have tensor outputs > 2D and want to operate softmax on other dimensionalities. You can change this easily in the second option
Question: In a post I posted yesterday, I accidentally found changing the __qualname__ of a function has an unexpected effect on pickle. By running more tests, I found that when pickling a function, pickle does not work in the way I thought, and changing the __qualname__ of the function has a real effect on how pickle behaves. The snippets below are tests I ran, import pickle from sys import modules # a simple function to pickle def hahaha(): return 1 print('hahaha',hahaha,'\n') # change the __qualname__ of function hahaha hahaha.__qualname__ = 'sdfsdf' print('set hahaha __qualname__ to sdfsdf',hahaha,'\n') # make a copy of hahaha setattr(modules['__main__'],'abcabc',hahaha) print('create abcabc which is just hahaha',abcabc,'\n') try: pickle.dumps(hahaha) except Exception as e: print('pickle hahaha') print(e,'\n') try: pickle.dumps(abcabc) except Exception as e: print('pickle abcabc, a copy of hahaha') print(e,'\n') try: pickle.dumps(sdfsdf) except Exception as e: print('pickle sdfsdf') print(e)  As you can see by running the snippets, both hahaha and abcabc cannot be pickled because of the exception: Can't pickle <function sdfsdf at 0x7fda36dc5f28>: attribute lookup sdfsdf on __main__ failed. I'm really confused by this exception,  What does pickle look for when it pickles a function? Although the __qualname__ of hahaha was changed to 'sdfsdf', the function hahaha as well as its copy abcabc is still callable in the session (as they are in dir(sys.modules['__main__'])), then why pickle cannot pickle them?  What is the real effect of changing the __qualname__ of a function? I understand by changing the __qualname__ of hahaha to 'sdfsdf' won't make sdfsdf callable, as it won't show up in dir(sys.modules['__main__']). However, as you can see by running the snippets, after changing the __qualname__ of hahaha to 'sdfsdf', the object hahaha as well as its copy abcabc has changed to something like <function sdfsdf at 'some_address'>. What is the difference between the objects in sys.modules['__main__'] and <function sdfsdf at 'some_address'>?  

Answer: Pickling of function objects is defined in the save_global method in pickle.py: First, the name of the function is retrieved via __qualname__: name = getattr(obj, '__qualname__', None)  Afterwards, after retrieving the module, it is reimported: __import__(module_name, level=0) module = sys.modules[module_name]  This freshly imported module is then used to look up the function as an attribute: obj2, parent = _getattribute(module, name)  obj2 would now be a new copy of the function, but since sdfsdf doesn't exist in this module, pickling fails here.  You can make this work, but you have to be consistent: >>> import sys >>> import pickle >>> def hahaha(): return 1 >>> hahaha.__qualname__ = "sdfsdf" >>> setattr(sys.modules["__main__"], "sdfsdf", hahaha) >>> pickle.dumps(hahaha) b'\x80\x04\x95\x17\x00\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\x06sdfsdf\x94\x93\x94.'
Question: I was reading the description of the two from the python doc:  spawn The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run the process objects run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver. [Available on Unix and Windows. The default on Windows and macOS.] fork The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic. [Available on Unix only. The default on Unix.]  And my question is:  is it that the fork is much quicker 'cuz it does not try to identify which resources to copy? is it that, since fork duplicates everything, it would "waste" much more resources comparing to spawn()? 

Answer:   is it that the fork is much quicker 'cuz it does not try to identify which resources to copy?   Yes, it's much quicker. The kernel can clone the whole process and only copies modified memory-pages as a whole. Piping resources to a new process and booting the interpreter from scratch is not necessary.   is it that, since fork duplicates everything, it would "waste" much more resources comparing to spawn()?   Fork on modern kernels does only "copy-on-write" and it only affects memory-pages which actually change. The caveat is that "write" already encompasses merely iterating over an object in CPython. That's because the reference-count for the object gets incremented. If you have long running processes with lots of small objects in use, this can mean you waste more memory than with spawn. Anecdotally I recall Facebook claiming to have memory-usage reduced considerably with switching from "fork" to "spawn" for their Python-processes.
Question: while pulling from git hub i lost my secret key which i have updated. is there any way to obtain secret key for the same project. while pulling from git hub i lost my secret key which i have updated. is there any way to obtain secret key for the same project.

Answer: run : python manage.py shell write and enter the following lines sequentially:  from django.core.management.utils import get_random_secret_key print(get_random_secret_key()) exit()  copy this secret_key to your settings.py SECRET_KEY. And reload this server. If it will not work, refresh the page with ctrl+shift+r, delete cache. If it will not work again, try to remove all rows from django_session table where in your database. My English skills are not good, sorry about that.
Question: I am facing the following error using SQLAlchemy: Unconsumed column names: company I want to insert data for 1 specific column, and not all columns in the table: INSERT INTO customers (company) VALUES ('sample name'); My code: engine.execute(table('customers').insert().values({'company': 'sample name'}))  Create Table: 'CREATE TABLE `customers` ( `id` int unsigned NOT NULL AUTO_INCREMENT, `company` varchar(255) DEFAULT NULL, `first_name` varchar(255) DEFAULT NULL, `last_name` varchar(255) DEFAULT NULL, `phone` varchar(255) DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `id_UNIQUE` (`id`), UNIQUE KEY `company_UNIQUE` (`company`) ) ENGINE=InnoDB AUTO_INCREMENT=63 DEFAULT CHARSET=utf8' 

Answer: After hours of frustration, I was able to test a way that I think works for my use case. As we know, you can insert to specific columns, or all columns in a table. In my use case, I dynamically need to insert to the customers table, depending on what columns a user has permissions to insert to. I found that I needed to define all columns in the table() method of sqlalchemy, but I can pass in whatever columns and values that I need dynamically to the values() method. Final code: engine.execute(table('customers', column('company'), column('first_name'), column('last_name'), column('email'), column('phone')).insert().values({'company': 'sample name'}))
Question: Is there a way to read data from a pandas DataFrame and construct a tree using anytree? Parent Child A A1 A A2 A2 A21  I can do it with static values as follows. However, I want to automate this by reading the data from a pandas DataFrame with anytree. >>> from anytree import Node, RenderTree >>> A = Node("A") >>> A1 = Node("A1", parent=A) >>> A2 = Node("A2", parent=A) >>> A21 = Node("A21", parent=A2)  Output is A ├── A1 └── A2 └── A21  This question AND especially the ANSWER has been adopted, copied really, from: Read data from a file and create a tree using anytree in python Many thanks to @Fabien N

Answer: Create nodes first if not exist, store their references in a dictionary nodes for further usage. Change parent when necessary for children. We can derive roots of the forest of trees by seeing what Parent values are not in Child values, since a parent is not a children of any node it won't appear in Child column. def add_nodes(nodes, parent, child): if parent not in nodes: nodes[parent] = Node(parent) if child not in nodes: nodes[child] = Node(child) nodes[child].parent = nodes[parent] data = pd.DataFrame(columns=["Parent","Child"], data=[["A","A1"],["A","A2"],["A2","A21"],["B","B1"]]) nodes = {} # store references to created nodes # data.apply(lambda x: add_nodes(nodes, x["Parent"], x["Child"]), axis=1) # 1-liner for parent, child in zip(data["Parent"],data["Child"]): add_nodes(nodes, parent, child) roots = list(data[~data["Parent"].isin(data["Child"])]["Parent"].unique()) for root in roots: # you can skip this for roots[0], if there is no forest and just 1 tree for pre, _, node in RenderTree(nodes[root]): print("%s%s" % (pre, node.name))  Result: A ├── A1 └── A2 └── A21 B └── B1  Update printing a specific root: root = 'A' # change according to usecase for pre, _, node in RenderTree(nodes[root]): print("%s%s" % (pre, node.name))
Question: I have a program that outputs arrays. For example: [[0, 1, 0], [0, 0, 0], [1, 3, 3], [2, 4, 4]]  I would like to turn these arrays into a dataframe using pandas. However, when I do the values become row values like this:  As you can see each array within the overall array becomes its own row. I would like each array within the overall array to become its own column with a column name. Furthermore, in my use case, the number of arrays within the array is variable. There could be 4 arrays or 70 which means there could be 4 columns or 70. This is problematic when it comes to column names and I was wondering if there was anyway to auto increment column names in python. Check out my attempt below and let me know how I can solve this. My desired outcome is simply to make each array within the overall array into its own column instead of row and to have titles for the column that increment with each additional array/column. Thank you so much. Need help. Please respond! frame = [[0, 1, 0], [0, 0, 0], [1, 3, 3], [2, 4, 4]] numpy_data= np.array(frame) df = pd.DataFrame(data=numpy_data, columns=["column1", "column2", "column3"]) print(frame) print(df) 

Answer: A possible solution could be transposing and renaming the columns after transforming the numpy array into a dataframe. Here is the code: import numpy as np import pandas as pd frame = [[0, 1, 0], [0, 0, 0], [1, 3, 3], [2, 4, 4]] numpy_data= np.array(frame) #transposing later df = pd.DataFrame(data=numpy_data).T #creating a list of columns using list comprehension without specifying number of columns df.columns = [f'mycol{i}' for i in range(0,len(df.T))] print(df)  Output:  mycol0 mycol1 mycol2 mycol3 0 0 0 1 2 1 1 0 3 4 2 0 0 3 4  Same code for 11 columns: import numpy as np import pandas as pd frame = [[0, 1, 0], [0, 0, 0], [1, 3, 3], [2, 4, 4], [5, 2, 2], [6,7,8], [8,9,19] , [10,2,4], [2,6,5], [10,2,5], [11,2,9]] numpy_data= np.array(frame) df = pd.DataFrame(data=numpy_data).T df.columns = [f'mycol{i}' for i in range(0,len(df.T))] print(df)   mycol0 mycol1 mycol2 mycol3 mycol4 mycol5 mycol6 mycol7 mycol8 mycol9 mycol10 0 0 0 1 2 5 6 8 10 2 10 11 1 1 0 3 4 2 7 9 2 6 2 2 2 0 0 3 4 2 8 19 4 5 5 9
Question: I want the 5th subplot to be in the centre of the two columns in the third row. (I have tried doing that by adding the domain argument). Here is the code to reproduce it- import pandas as pd import plotly.graph_objects as go from plotly.subplots import make_subplots continent_df = pd.read_csv('https://raw.githubusercontent.com/vyaduvanshi/helper-files/master/continent.csv') temp_cont_df = pd.pivot_table(continent_df, index='continent', aggfunc='last').reset_index() fig = make_subplots(rows=3, cols=2, specs=[[{'type':'pie'},{'type':'pie'}],[{'type':'pie'},{'type':'pie'}], [{'type':'pie'},{'type':'pie'}]]) fig.add_pie(labels=continent_df.continent, values=continent_df.new_cases, row=1,col=1) fig.add_pie(labels=continent_df.continent, values=continent_df.new_deaths, row=1,col=2) fig.add_pie(labels=continent_df.continent, values=continent_df.new_recovered, row=2,col=1) fig.add_pie(labels=continent_df.continent, values=continent_df.new_tests, row=2,col=2) fig.add_pie(labels=temp_cont_df.continent, values=temp_cont_df.active_cases, row=3,col=1,domain={'x':[0.25,0.75],'y':[0,0.33]})   If I do not include the 6th plot in the specs argument, it throws an error.

Answer: You can achieve this through a correct setup of domain. Here's an example that will have a figure in each of the four corners and one figure in the middle. Plot  Complete code: import plotly import plotly.offline as py import plotly.graph_objs as go labels = ['Oxygen','Hydrogen','Carbon_Dioxide','Nitrogen'] values = [4500,2500,1053,500] domains = [ {'x': [0.0, 0.33], 'y': [0.0, 0.33]}, {'x': [0.33, 0.66], 'y': [0.33, 0.66]}, {'x': [0.0, 0.33], 'y': [0.66, 1.0]}, {'x': [0.66, 1.00], 'y': [0.0, 0.33]}, {'x': [0.66, 1.0], 'y': [0.66, 1.00]}, ] traces = [] for domain in domains: trace = go.Pie(labels = labels, values = values, domain = domain, hoverinfo = 'label+percent+name') traces.append(trace) layout = go.Layout(height = 600, width = 600, autosize = False, title = 'Main title') fig = go.Figure(data = traces, layout = layout) #py.iplot(fig, show_link = False) fig.show()
Question: I have a pandas dataframe with name of variables, the values for each and the count (which shows the frequency of that row): df = pd.DataFrame({'var':['A', 'B', 'C'], 'value':[10, 20, 30], 'count':[1,2,3]}) var value count A 10 1 B 20 2 C 30 3  I want to use count to get an output like this: var value A 10 B 20 B 20 C 30 C 30 C 30  What is the best way to do that?

Answer: You can use index.repeat: i = df.index.repeat(df['count']) d = df.loc[i, :'value'].reset_index(drop=True)    var value 0 A 10 1 B 20 2 B 20 3 C 30 4 C 30 5 C 30