eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: Hello everyone and thank you in advance, I am trying to get a all the words in the following list except for "motiu" and "diomar" using regex and python: amfora difamador difamar dimorf dofi fada far farao farda fiar fiord fira firar firma for motiu diomar  The word must not contain a letter outside the list [diomarf], but it must contain an "f" I don't know much about regex...I have tried with some, they are getting more complex but I haven't got the solution yet. Some of the expressions I have tried with are: > (?:.*f)(?:.*[diomarf]) > (?:.*[diomarf])(?:.*f) > (?:((?:f)+)(?:[diomarf])*) > (?:((?:[diomarf])+)(?:f)*) > (?:((?:[diomarf])*)((?:f)+)) > (?:(((?:f)+)((?:[diomarf])*))) > (?:((?:f)+((?:[diomarf])*)))  The expression with which I think I got the closest result is:  (?:(((?:f)+)((?:[diomarf])*)))  But it only checks from the first f of the word, for example, for "dimorf" I am only getting the last "f"

Answer: ^f[diomarf]*$|^[diomarf]*f[diomarf]*$|^[diomarf]*f$  demo Explanation: ^f[diomarf]*$ : A string either starts with f and then has any number of characters from this list [diomarf] (including 0!) OR ^[diomarf]*f[diomarf]*$ the string has f somewhere in the middle OR ^[diomarf]*f$ f at the end The previous solution I proposed fails when disallowed characters are added to the end of the string (for example diomarfg). Old solution for reference: (?=[diomarf]).*f.*  demo here Explanation: (?=[diomarf]) - use positive lookahead to assert that at any point in the string one of the allowed letters is matched. .*f.* - make sure that the letter f is somewhere in the string.
Question: I would like to improve my bitcoin dataset but I found that the date is not sorted in the right way and want to show only the month and year. How can I do it? data = Bitcoin_Historical['Price'] Date1 = Bitcoin_Historical['Date'] train1 = Bitcoin_Historical[['Date','Price']] #Setting the Date as Index train2 = train1.set_index('Date') train2.sort_index(inplace=True) cols = ['Price'] train2 = train2[cols].apply(lambda x: pd.to_numeric(x.astype(str) .str.replace(',',''), errors='coerce')) print (type(train2)) print (train2.head()) plt.figure(figsize=(15, 5)) plt.plot(train2) plt.xlabel('Date', fontsize=12) plt.xlim(0,20) plt.ylabel('Price', fontsize=12) plt.title("Closing price distribution of bitcoin", fontsize=15) plt.gcf().autofmt_xdate() plt.show()  The result shows picture below:  It's not ordered and shows all dates. I would like to order by month+year and show only the month name+year. How can that be done? Example of Data:  Thank you

Answer: I've made the following edits to your code:  converted the column Date column as datetime type cleaned up the Price column and converting to float removed the line plt.xlim(0,20) which is causing the output to display 1970 used alternative way to plot, so that the x-axis can be formatted to get monthly tick marks, more info here  Please try the code below: import pandas as pd import matplotlib.pyplot as plt import matplotlib.dates as mdates pd.options.mode.chained_assignment = None Bitcoin_Historical = pd.read_csv('data.csv') train1 = Bitcoin_Historical[['Date','Price']] train1['Date'] = pd.to_datetime(train1['Date'], infer_datetime_format=True, errors='coerce') train1['Price'] = train1['Price'].str.replace(',','').str.replace(' ','').astype(float) train2 = train1.set_index('Date') #Setting the Date as Index train2.sort_index(inplace=True) print (type(train2)) print (train2.head()) ax = train2.plot(figsize=(15, 5)) ax.xaxis.set_major_locator(mdates.MonthLocator(interval=1)) ax.xaxis.set_major_formatter(mdates.DateFormatter('%Y-%b')) plt.xlabel('Date', fontsize=12) plt.ylabel('Price', fontsize=12) plt.title("Closing price distribution of bitcoin", fontsize=15) plt.show()  Output
Question: I have a JSON file and need to update with new key value pair. cuurent json: [{'Name': 'AMAZON', 'Type': 'Web', 'eventTimeEpoch': 1611667194}]  I need to add a location parameter and update it as "USA".But when try to update it with below code it append to location parameter with value to end. Like below. [{'Name': 'AMAZON', 'Type': 'Web', 'eventTimeEpoch': 1611667194, 'location': 'USA'}]  How can I add this location parameter after the Name. Expected output: [{'Name': 'AMAZON', 'location': 'USA', 'Type': 'Web', 'eventTimeEpoch': 1611667194 }]  Current code: filename='test.json' jsonFile = open(filename, "r") # Open the JSON file for reading data = json.load(jsonFile) jsonFile.close() data[0]["location"] = "USA" data 

Answer: First off, your current JSON file is not correctly formatted. JSON needs double quotes not single quotes. While the order of the inserted key-value pairs is guaranteed to be preserved in Python 3.7 above, you don't have any option to "insert" to specific location inside a dictionary.(like list for example.) And people usually don't count on the order of the keys when working with JSON files. You get your values by "keys" anyway. With that being said, you can do something like: import json with open("test.json") as f: data = json.load(f) print(data) new_d = {"Name": data[0].pop("Name"), "location": "USA", **data[0]} print(new_d)  This way we created a new dictionary with the desired order. Since "location" key is near to the start of the items, we pop first key-value pair, then insert the "location" key, then unpack the rest with ** operator.
Question: Let ll be a list of lists, and tt a tuple of tuples Input: ll = [["a1","a2"],["b1","b2"],["c1","c2"]] Desired output: tt = (("a1","b1","c1"),("a2","b2","c2")) I have managed to solve it for a list of two-element lists, meaning that the internal list only contained two elements each. def list_of_list_to_tuple_of_tuple(ll): first_elements = [i[0] for i in ll] second_elements = [i[1] for i in ll] new_list = [] new_list.append(tuple(first_elements)) new_list.append(tuple(second_elements)) return tuple(new_list) ll = [["a1","a2"],["b1","b2"],["c1","c2"]] list_of_list_to_tuple_of_tuple(ll)  Now, the questions are:  Is there any other method to easily accomplish what I have done?  Is there any method to easily generalize this algorithm if we have a list of 3 internal lists and each internal list containing n elements? For example:   Input: ll = [["a1","a2","a3",..."an"],["b1","b2","b3",..."bn"],["c1","c2","c3",..."cn"]] Desired Output: tt = (("a1","b1","c1"),("a2","b2","c2"),("a3","b3","c3"),...,("an","bn","cn"))

Answer: Try this one-liner - tuple(zip(*l))  Example 1 l = [["a1","a2"], ["b1","b2"], ["c1","c2"]] tuple(zip(*l))  (('a1', 'b1', 'c1'), ('a2', 'b2', 'c2'))  Example 2 l2 = [["a1","a2","a3","an"], ["b1","b2","b3","bn"], ["c1","c2","c3","cn"]] tuple(zip(*l2))  (('a1', 'b1', 'c1'), ('a2', 'b2', 'c2'), ('a3', 'b3', 'c3'), ('an', 'bn', 'cn'))  EXPLANATION  The unpacking operator allows you to unpack the list into the sublists, and passes them as individual parameters to zip, as it expects the same. The zip combines the first, second, third ... nth respective elements of each sublist into n tuples object    The tuple converts this zip object converts the overall zip object to a tuple.   Bonus Intuitively, this operation resembles taking a transpose of a matrix. This can be seen easily if you convert your list of lists to a numpy array and then take a transpose. import numpy as np l = [["a1","a2"],["b1","b2"],["c1","c2"]] arr = np.array(l) transpose = arr.T transpose  array([['a1', 'b1', 'c1'], ['a2', 'b2', 'c2']], dtype='<U2')
Question: I have a generic base class and I want to be able to inspect the provided type for it. My approach was using typing.get_args which works like so: from typing import Generic, Tuple, TypeVarTuple, get_args T = TypeVarTuple("T") class Base(Generic[*T]): values: Tuple[*T] Example = Base[int, str] print(get_args(Example)) # (<class 'int'>, <class 'str'>)  But when I'm inheriting the class, I'm getting an empty list of parameters like so: class Example2(Base[int, str]): pass print(get_args(Example2)) # ()  What I actually need is to know what types are expected for the values property. I might have the wrong approach but I've also tried to use typing.get_type_hints which seems to just return Tuple[*T] as the type. So how can I get the typed parameters? Edit: I need to know the types of the class, not the object.

Answer: Use get_args with __orig_bases__: print(get_args(Example2.__orig_bases__[0])) # prints "(<class 'int'>, <class 'str'>)"  For convenience, you can store the generic type parameters in the __init_subclass__ hook: from typing import Generic, TypeVarTuple, get_args T = TypeVarTuple("T") class Base(Generic[*T]): values: tuple[*T] type_T: tuple[type, ...] def __init_subclass__(cls) -> None: cls.type_T = get_args(cls.__orig_bases__[0]) # type: ignore class Example2(Base[int, str]): pass print(Example2.type_T) # prints "(<class 'int'>, <class 'str'>)"
Question: I have a boolean sparse matrix that I represent with row indices and column indices of True values. import numpy as np import jax from jax import numpy as jnp N = 10000 M = 1000 X = np.random.randint(0, 100, size=(N, M)) == 0 # data setup rows, cols = np.where(X == True) rows = jax.device_put(rows) cols = jax.device_put(cols)  I want to get a column slice of the matrix like X[:, 3], but just from rows indices and column indices. I managed to do that by using jnp.isin like below, but the problem is that this is not JIT compatible because of the data-dependent shaped array rows[cols == m]. def not_jit_compatible_slice(rows, cols, m): return jnp.isin(jnp.arange(N), rows[cols == m])  I could make it JIT compatible by using jnp.where in the three-argument form, but this operation is much slower than the previous one. def jit_compatible_but_slow_slice(rows, cols, m): return jnp.isin(jnp.arange(N), jnp.where(cols == m, rows, -1))  Is there any fast and JIT compatible solution to acheive the same output?

Answer: You can do a bit better than the first answer by using the mode argument of set() to drop out-of-bound indices, eliminating the final slice: out = jnp.zeros(N, bool).at[jnp.where(cols==3, rows, N)].set(True, mode='drop')
Question: I would like to understand how to use pyparsing to parse something like a nested Python list. This is a question to understand pyparsing. Solutions that circumvent the problem because the list of the example might look like JSON or Python itself should not prevent the usage of pyparsing. So before people start throwing json and literal_eval at me let's consider a string and result that looks like this: Input: {1,2,3,{4,5}} Expected Output (Python list): [1,2,3,[4,5]]  I currently have this code but the output does not parse the nested list import pyparsing print( pyparsing.delimited_list( pyparsing.Word(pyparsing.nums) | pyparsing.nested_expr("{", "}") ) .parse_string("{1,2,3,{4,5}}") .as_list() ) # [['1,2,3,', ['4,5']]]  There is pretty much the same question here already but this one was circumvented by using json parsing: Python parse comma seperated nested brackets using Pyparsing

Answer: Thanks to the answer from Xiddoc I was able to slightly adjust the answer to also work when the expression starts with a list (no idea why the solution with nested_expr does not work) import pyparsing as pp expr = pp.Forward() group_start, group_end = map(pp.Suppress, r"{}") number = pp.Word(pp.nums).setParseAction(lambda s, l, t: int(t[0])) nested_list = pp.Group(group_start + expr[...] + group_end) expr <<= pp.delimited_list(number | nested_list) print(expr.parse_string(r"{{1},2,3,{4,5}}", parse_all=True).as_list()[0]) # [[1],2,3,[4,5]]
Question: I have a df that has a column 'Time' in seconds: Time 1 2 3 4  I want the user to input the date with a timestamp (eg format: 25/09/2022 12:30:00). Then, I need to add a new column 'DateTime' which combines the user input datetime with my 'Time' column. The 'DateTime' column should look like this: DateTime 25/09/2022 12:30:01 25/09/2022 12:30:02 25/09/2022 12:30:03 25/09/2022 12:30:04  I managed to do this in python, where the user input is on the terminal, however, I would like to have this in Streamlit. From the documentation, there is currently no possibility to input date with a timestamp in Streamlit, unless you enter them separately, as follows: start_date = st.date_input('Enter start date', value=datetime.datetime(2019,7,6)) start_time = st.time_input('Enter start time', datetime.time(8, 45))  So, this gives the user the possibility to enter the date and time separately, however I don't know how to derive my 'DateTime' column and add it to the df. Appreciate any advice on how to accomplish this.

Answer: You can use the .combine() function from pandas to combine your start_date with start_time, after you have accomplished that. Make a new df named DateTime and loop through your Time df to concatenate the seconds to DateTime, after the concatenation, format the DateTime. You can then drop Time column after haven looped through it. Example: # Your df that contains "Time" column df = pd.DataFrame({"Time":[1, 2, 3, 4]}) start_date = st.date_input('Enter start date', value=datetime.datetime(2019,7,6)) start_time = st.time_input('Enter start time', datetime.time(8, 45)) start_datetime = datetime.datetime.combine(start_date, start_time) df["DateTime"] = [start_datetime + datetime.timedelta(seconds=time) for time in df["Time"]] df["DateTime"] = [date.strftime("%d/%m/%Y %H:%M:%S") for date in df["DateTime"]] df = df.drop(columns=["Time"]) st.dataframe(df)  Output:  DateTime 0 06/07/2019 08:45:01 1 06/07/2019 08:45:02 2 06/07/2019 08:45:03 3 06/07/2019 08:45:04
Question: I am using Python to convert Pandas df to .xlsx (in Plotly-Dash app.). All working well so far but with this warning tho: "FutureWarning: save is not part of the public API, usage can give unexpected results and will be removed in a future version" How should I modify the code below in order to keep its functionality and stability in future? Thanks!  writer = pd.ExcelWriter("File.xlsx", engine = "xlsxwriter") workbook = writer.book df.to_excel(writer, sheet_name = 'Sheet', index = False) writer.save() 

Answer: just replace save with close.  writer = pd.ExcelWriter("File.xlsx", engine = "xlsxwriter") workbook = writer.book df.to_excel(writer, sheet_name = 'Sheet', index = False) writer.close()
Question: Given an existing networkx graph import networkx as nx import numpy as np np.random.seed(123) graph = nx.erdos_renyi_graph(5, 0.3, seed=123, directed=True) nx.draw_networkx(graph)   or import networkx as nx G = nx.path_graph(4) nx.spring_layout(G) nx.draw_networkx(G)   how can you draw a red circle on top of (in the same position as) one of the nodes, like the node labeled 1?

Answer: To be able to draw a networkx graph, each node needs to be assigned a position. By default, nx.spring_layout() is used to calculate positions when calling nx.draw_networkx(), but these positions aren't stored. They are recalculated each time the function is drawn, except when the positions are explicitly added as a parameter. Therefore, you can calculate these positions beforehand, and then use these to plot circles: import matplotlib.pyplot as plt from matplotlib.colors import to_rgba import networkx as nx import numpy as np np.random.seed(123) graph = nx.erdos_renyi_graph(5, 0.3, seed=123, directed=True) pos = nx.spring_layout(graph) nx.draw_networkx(graph, pos=pos) ax = plt.gca() for node_id, color in zip([1, 4], ['crimson', 'limegreen']): ax.add_patch(plt.Circle(pos[node_id], 0.15, facecolor=to_rgba(color, alpha=0.2), edgecolor=color)) ax.set_aspect('equal', 'datalim') # equal aspect ratio is needed to show circles undistorted plt.show()
Question: I have an array that has pairs of numbers representing row, col values in a model domain. I am trying to add the layer value to have a list of lay, row, col. I have an array rowcol: array([(25, 65), (25, 66), (25, 67), (25, 68), (26, 65), (26, 66), (26, 67), (26, 68), (26, 69), (27, 66), (27, 67), (27, 68), (27, 69), (28, 67), (28, 68)], dtype=object)  and I want to add an 8 to each pair so it looks like array([(8, 25, 65), (8, 25, 66), (8, 25, 67), (8, 25, 68), (8, 26, 65), (8, 26, 66), (8, 26, 67), (8, 26, 68), (8. 26, 69), (8, 27, 66), (8, 27, 67), (8, 27, 68), (8, 27, 69), (8, 28, 67), (8, 28, 68)], dtype=object)  I created a new array (layer) that was the same length as rowcol and zipped the 2 with: layrowcol = list(zip(layer, rowcol)) and ended up with: [(8, (25, 65)), (8, (25, 66)), (8, (25, 67)), (8, (25, 68)), (8, (26, 65)), (8, (26, 66)), (8, (26, 67)), (8, (26, 68)), (8, (26, 69)), (8, (27, 66)), (8, (27, 67)), (8, (27, 68)), (8, (27, 69)), (8, (28, 67)), (8, (28, 68))]  So it sort of worked and yet didn't quite. Is there a way to combine them and leave out the unwanted parentheses or some better way to add the layer value to each pair without using zip(). Any help is appreciated.

Answer: You can use numpy.insert. >>> import numpy as np >>> a = np.array([(25, 65), (25, 66), (25, 67), (25, 68), (26, 65), (26, 66),(26, 67), (26, 68), (26, 69), (27, 66), (27, 67), (27, 68),(27, 69), (28, 67), (28, 68)], dtype=object) >>> b = np.insert(a, 0, 8, axis=1)  Output: array([[8, 25, 65], [8, 25, 66], [8, 25, 67], [8, 25, 68], [8, 26, 65], [8, 26, 66], [8, 26, 67], [8, 26, 68], [8, 26, 69], [8, 27, 66], [8, 27, 67], [8, 27, 68], [8, 27, 69], [8, 28, 67], [8, 28, 68]], dtype=object)  If you want back to the list of tuples. >>> list(map(tuple, b)) [(8, 25, 65), (8, 25, 66), (8, 25, 67), (8, 25, 68), (8, 26, 65), (8, 26, 66), (8, 26, 67), (8, 26, 68), (8, 26, 69), (8, 27, 66), (8, 27, 67), (8, 27, 68), (8, 27, 69), (8, 28, 67), (8, 28, 68)]
Question: How to minmax normalize in the most efficient way, a XD-numpy array in "columns" of each 2D matrix of the array. For example with a 3D-array : a = np.array([[[ 0, 10], [ 20, 30]], [[ 40, 50], [ 60, 70]], [[ 80, 90], [100, 110]]])  into the normalized array : b = np.array([[[0., 0.], [1., 1.]], [[0., 0.], [1., 1.]], [[0., 0.], [1., 1.]]]) 

Answer: With sklearn.preprocessing.minmax_scale + numpy.apply_along_axis single applying: from sklearn.preprocessing import minmax_scale a = np.array([[[0, 10], [20, 30]], [[40, 50], [60, 70]], [[80, 90], [100, 110]]]) a_scaled = np.apply_along_axis(minmax_scale, 1, a)   # a_scaled [[[0. 0.] [1. 1.]] [[0. 0.] [1. 1.]] [[0. 0.] [1. 1.]]]
Question: I have multiple classes similar to the following: class Weather(Base): __tablename__ = "Weather" id = Column(Integer, primary_key=True, nullable=False, autoincrement=True) temperature = Column(Integer) humidity = Column(Integer) wind_speed = Column(Float) wind_direction = Column(String)  I want to add a method df() that returns me the Pandas dataframe of that table. I know I can write it like this: class Weather(Base): __tablename__ = "Weather" id = Column(Integer, primary_key=True, nullable=False, autoincrement=True) temperature = Column(Integer) humidity = Column(Integer) wind_speed = Column(Float) wind_direction = Column(String) @staticmethod def df(): with engine.connect() as conn: return pd.read_sql_table(Weather.__tablename__ , conn)  But I want to implement this for every table. I guess if I can extend the Base class with this method I should be able to implement it once and use it in every class. Everything I have tried has failed because I do not have access to __tablename__  attribute. SOLUTION I ended up with a mix of both answers. I have used the first method proposed by @snakecharmerb (it allows to introduce the change without modifying the rest of the code) with the @classmethod proposed by @RomanPerekhrest (which is the bit I was missing). class MyBase: __tablename__ = None @classmethod def df(cls): with engine.connect() as conn: return pd.read_sql_table(cls.__tablename__ , conn) Base = declarative_base(cls=MyBase) 

Answer: Declare an auxiliary class (say DfBase) with classmethod df(cls) having the desired behavior. Then each derived class will access its __tablename__ attribute seamlessly via cls object which refers to the derived class itself. class DfBase: __tablename__ = None @classmethod def df(cls): with engine.connect() as conn: return pd.read_sql_table(cls.__tablename__ , conn) class Weather(Base, DfBase): __tablename__ = "Weather" ...
Question: MyPy thinks it has to check for Python <3.8 when instead it should use 3.10 As you can see, Python3.10 is active (myvenv) gitpod /workspace/myfolder (mybranch) $ python --version Python 3.10.7  however mypy think its <3.8? (myvenv) gitpod /workspace/myfolder (mybranch) $ mypy -p my_folder_with_code  /workspace/.pyenv_mirror/poetry/virtualenvs/myenv/lib/python3.10/site-packages/numpy/__init__.pyi:641: error: Positional-only parameters are only supported in Python 3.8 and greater Found 1 error in 1 file (errors prevented further checking)  even mypy --python-version 3.10 -p my_folder_with_code produces the same error This happens only in this platform (gitpod). On other devices it runs fine (so no error in code) I googled around but did found what i'm looking for... can somebody help?

Answer: Ok, i found it out miself. Apparently it was a bug in mypy. Update MyPy and it works again pip install -U mypy Or in my case poetry update mypy
Question: I apply this function def calculate_recency_for_one_column(column: pd.Series) -> int: """Returns the inverse position of the last non-zero value in a pd.Series of numerics. If the last value is non-zero, returns 1. If all values are non-zero, returns 0.""" non_zero_values_of_col = column[column.astype(bool)] if non_zero_values_of_col.empty: return 0 return len(column) - non_zero_values_of_col.index[-1]  to all columns of this example dataframe df = pd.DataFrame(np.random.binomial(n=1, p=0.001, size=[1000000]).reshape((1000,1000)))   by using df.apply(lambda column: calculate_recency_for_one_column(column),axis=0)  The result is: 0 436 1 0 2 624 3 0 ... 996 155 997 715 998 442 999 163 Length: 1000, dtype: int64  Everything works fine, but my programm has to do this operation often, so I need a more efficient alternative. Does anybody have an idea how to make this faster? I think calculate_recency_for_one_column() is efficient enough and the df.apply() has the most potential for improvement. Here a as benchmark (100 reps): >> timeit.timeit(lambda: df.apply(lambda column: calculate_recency_for_one_column(column),axis=0), number=100) 14.700050864834338  Update Mustafa's answer: >> timeit.timeit(lambda: pd.Series(np.where(df.eq(0).all(), 0, len(df) - df[::-1].idxmax())), number=100) 0.8847485752776265  padu's answer: >> timeit.timeit(lambda: df.apply(calculate_recency_for_one_column_numpy, raw=True, axis=0), number=100) 0.8892530500888824 

Answer: You can treat columns not as Series objects but as numpy arrays. To do this, simply specify the raw=True parameter in the apply method. also need to slightly change the original function. import time import numpy as np import pandas as pd def calculate_recency_for_one_column(column: np.ndarray) -> int: """Returns the inverse position of the last non-zero value in a np.ndarray of numerics. If the last value is non-zero, returns 1. If all values are non-zero, returns 0.""" non_zero_values_of_col = np.nonzero(column)[0] if not non_zero_values_of_col.any(): return 0 return len(column) - non_zero_values_of_col[-1] df = pd.DataFrame(np.random.binomial(n=1, p=0.001, size=[1000000]).reshape((1000,1000))) start = time.perf_counter() res = df.apply(calculate_recency_for_one_column, raw=True) print(f'time took {time.perf_counter() - start:.3f} s.')  Out: 0.005 s.