eshangj/stackoverflow_q_and_a_sample huggingface dataset:
Question: I have installed pytube to extract captions from some youtube videos. Both the following code give me the xml captions. from pytube import YouTube yt = YouTube('https://www.youtube.com/watch?v=4ZQQofkz9eE') caption = yt.captions['a.en'] print(caption.xml_captions)  and also as mentioned in the docs yt = YouTube('http://youtube.com/watch?v=2lAe1cqCOXo') caption = yt.captions.get_by_language_code('en') caption.xml_captions  But in both cases, I get the xml output and when use print(caption.generate_srt_captions())  I get an error like the following. Can you help on how to extract the srt format? KeyError ~/anaconda3/envs/myenv/lib/python3.6/site-packages/pytube/captions.py in generate_srt_captions(self) 49 recompiles them into the "SubRip Subtitle" format. 50 """ 51 return self.xml_caption_to_srt(self.xml_captions) 52 53 @staticmethod ~/anaconda3/envs/myenv/lib/python3.6/site-packages/pytube/captions.py in xml_caption_to_srt(self, xml_captions) 81 except KeyError: 82 duration = 0.0 83 start = float(child.attrib["start"]) 84 end = start + duration 85 sequence_number = i + 1 # convert from 0-indexed to 1. KeyError: 'start' 

Answer: This is a bug in the library itself. Everything below is done in pytube 11.01. In the captions.py file on line 76 replace: for i, child in enumerate(list(root)):  to: for i, child in enumerate(list(root.findall('body/p'))):  Then on line 83, replace: duration = float(child.attrib["dur"])  to: duration = float(child.attrib["d"])  Then on line 86, replace: start = float(child.attrib["start"])  to: start = float(child.attrib["t"])  If only the number of lines and time will be displayed but no subtitle text, replace line 77: text = child.text or ""  to: text = ''.join(child.itertext()).strip() if not text: continue  It worked for me, python 3.9, pytube 11.01. Good luck!
Question: I've loaded some data and modified one column in the dataframe and would like to update the DB to reflect the changes. I tried: db.session.query(sqlTableName).update({sqlTableName.sql_col_name: pdDataframe.pd_col_name}) But that just wiped out the column in the database (set every value to '0', the default). I tried a few other dataformats with no luck. I'm guessing that there is something funky going on with datatypes that I've mixed up, or you just aren't allowed to update a column with a variable like this directly. I could do this with a loop but... that would be genuinely awful. Sorry for the basic question, after a long break from a project, my grasp of sqlalchemy has certainly waned.

Answer: For uploading the DataFrame to a temporary table and then performing an UPDATE you don't need to write the SQL yourself, you can have SQLAlchemy Core do it for you: import pandas as pd import sqlalchemy as sa def update_table_columns_from_df(engine, df, table_name, cols_to_update): metadata = sa.MetaData() main_table = sa.Table(table_name, metadata, autoload_with=engine) pk_columns = [x.name for x in main_table.primary_key.columns] df.to_sql("temp_table", engine, index=False, if_exists="replace") temp_table = sa.Table("temp_table", metadata, autoload_with=engine) with engine.begin() as conn: values_clause = {x: temp_table.columns[x] for x in cols_to_update} where_clause = sa.and_( main_table.columns[x] == temp_table.columns[x] for x in pk_columns ) conn.execute( main_table.update().values(values_clause).where(where_clause) ) temp_table.drop(engine) if __name__ == "__main__": test_engine = sa.create_engine( "postgresql+psycopg2://scott:tiger@192.168.0.199/test", echo=True, # (for demonstration purposes) ) with test_engine.begin() as test_conn: test_conn.exec_driver_sql("DROP TABLE IF EXISTS main_table") test_conn.exec_driver_sql( """\ CREATE TABLE main_table ( id1 integer NOT NULL, id2 integer NOT NULL, txt1 varchar(50), txt2 varchar(50), CONSTRAINT main_table_pkey PRIMARY KEY (id1, id2) ) """ ) test_conn.exec_driver_sql( """\ INSERT INTO main_table (id1, id2, txt1, txt2) VALUES (1, 1, 'foo', 'x'), (1, 2, 'bar', 'y'), (1, 3, 'baz', 'z') """ ) df_updates = pd.DataFrame( [ (1, 1, "new_foo", "new_x"), (1, 3, "new_baz", "new_z"), ], columns=["id1", "id2", "txt1", "txt2"], ) update_table_columns_from_df( test_engine, df_updates, "main_table", ["txt1", "txt2"] ) """SQL emitted: UPDATE main_table SET txt1=temp_table.txt1, txt2=temp_table.txt2 FROM temp_table WHERE main_table.id1 = temp_table.id1 AND main_table.id2 = temp_table.id2 """ df_result = pd.read_sql_query( "SELECT * FROM main_table ORDER BY id1, id2", test_engine ) print(df_result) """ id1 id2 txt1 txt2 0 1 1 new_foo new_x 1 1 2 bar y 2 1 3 new_baz new_z """
Question: Logger in PyTorch-Lightning prints information about the model to be trained (or evaluated) and the progress during the training, However, in my case I would like to hide all messages from the logger in order not to flood the output in Jupyter Notebook. I've looked into the API of the Trainer class on the official docs page https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#trainer-flags and it seems like there is no option to turn off the messages from the logger. There is a parameter log_every_n_steps which can be set to big value, but nevertheless, the logging result after each epoch is displayed. How can one disable the logging?

Answer: I am assuming that two things are particularly bothering you in terms of flooding output stream: One, The "weight summary":  | Name | Type | Params -------------------------------- 0 | l1 | Linear | 100 K 1 | l2 | Linear | 1.3 K -------------------------------- ...  Second, the progress bar: Epoch 0: 74%|███████████ | 642/1874 [00:02<00:05, 233.59it/s, loss=0.85, v_num=wxln]  PyTorch Lightning provided very clear and elegant solutions for turning them off: Trainer(progress_bar_refresh_rate=0) for turning off progress bar and Trainer(weights_summary=None) for turning off weight summary.
Question: I fine tuned the pretrained model here by freezing all layers except the classifier layers. And I saved weight file with using pytorch as .bin format. Now instead of loading the 400mb pre-trained model, is there a way to load the parameters of the just Classifier layer I retrained it? By the way, I know that I have to load the original pretrained model, I just don't want to load the entire fine tuned model. due to memory concerns. I can access the last layer's parameters from state_dict as below, but how can I save them in a separate file to use them later for less memory usage? model = PosTaggingModel(num_pos_tag=num_pos_tag) state_dict = torch.load("model.bin") print("state dictionary:",state_dict) with torch.no_grad(): model.out_pos_tag.weight.copy_(state_dict['out_pos_tag.weight']) model.out_pos_tag.bias.copy_(state_dict['out_pos_tag.bias'])  Here is the model class: class PosTaggingModel(nn.Module): def __init__(self, num_pos_tag): super(PosTaggingModel, self).__init__() self.num_pos_tag = num_pos_tag self.model = AutoModel.from_pretrained("dbmdz/bert-base-turkish-cased") for name, param in self.model.named_parameters(): if 'classifier' not in name: # classifier layer param.requires_grad = False self.bert_drop = nn.Dropout(0.3) self.out_pos_tag = nn.Linear(768, self.num_pos_tag) def forward(self, ids, mask, token_type_ids, target_pos_tag): o1, _ = self.model(ids, attention_mask = mask, token_type_ids = token_type_ids) bo_pos_tag = self.bert_drop(o1) pos_tag = self.out_pos_tag(bo_pos_tag) loss = loss_fn(pos_tag, target_pos_tag, mask, self.num_pos_tag) return pos_tag, loss  I don't know if this is possible but I'm just looking for a way to save and reuse the last layer's parameters, without the need for parameters of frozen layers. I couldn't find it in the documentation. Thanks in advance to those who will help.

Answer: You can do it like this import torch # creating a dummy model class Classifier(torch.nn.Module): def __init__(self): super(Classifier, self).__init__() self.first = torch.nn.Linear(10, 10) self.second = torch.nn.Linear(10, 20) self.last = torch.nn.Linear(20, 1) def forward(self, x): pass # Creating its object model = Classifier() # Extracting the layer to save to_save = model.last # Saving the state dict of that layer torch.save(to_save.state_dict(), './classifier.bin') # Recreating the object of that model model = Classifier() # Updating the saved layer of model model.last.load_state_dict(torch.load('./classifier.bin'))
Question: Using Python 3.8, Plotly 4.13. Within my scatterplot, I've added multiple vertical lines using add_vline() method in plotly. However I cannot add it to legend allowing me to turn on/off vertical lines. How can add vertical lines to the legend? Here is example of how I've created my plot: fig = go.Figure() fig.add_trace( go.Scatter(name="name added to legend", datas...) ) for dt in dates: fig.add_vline(x=dt, line_width=1, etc...)  outputting something like this:  all plots created using go.Scatter are added to legend however not vertical lines created by fig.add_vline.

Answer: The vertical line is just a decoration of the graph, not a graph object, so it is not included in the legend. So if you want to use the ON/OFF function in the legend, you can add a new one in go.Scatter and use it for the function to select in the legend. I modified the example from the official reference to create the code. import plotly.express as px import plotly.graph_objects as go df = px.data.stocks() dmax = df[['GOOG', 'AAPL', 'AMZN', 'FB', 'NFLX', 'MSFT']].values.max() dmin = df[['GOOG', 'AAPL', 'AMZN', 'FB', 'NFLX', 'MSFT']].values.min() fig = go.Figure() fig.add_trace(go.Scatter(x=df.date, y=df['AAPL'], mode='lines', name='AAPL')) fig.add_trace(go.Scatter(x=df.date, y=df['GOOG'], mode='lines', name='GOOG')) fig.add_trace(go.Scatter(x=[df.date[10],df.date[10]], y=[dmin,dmax], mode='lines', line=dict(color='green', width=2, dash='dash'), name='2018-03-12')) fig.show()
Question:  Please explain the behavior of the set in the image. I know that set is unordered but where are the other elements from the set a & b ?

Answer: True and 1 are the same: >>> True == 1 True >>>  Since sets can't have duplicate values, it only takes the one that appears first. You can see that if you convert True to int: >>> int(True) 1 >>>  The output is 1.
Question: I'm getting the error: TypeError: scatter() got an unexpected keyword argument 'trendline_options'  When trying to adjust the smoothing of the lowess tendline using plotly express. Here is my code for the graph: fig = px.scatter(dfg, x="Yr_Mnth", y="Episode_Count", color = "Target", labels={"Episode_Count": tally + " per Shift", "Target":"Target", "Yr_Mnth": "Date" }, trendline='lowess',trendline_options= dict(frac=0.1), title="Aggregate Behavior Data: " + patient + " - " + today) fig.update_xaxes(tickangle=45,) fig.update_layout(template = 'plotly_white',hovermode="x unified")  Dataset (dfg): Yr_Mnth Target Episode_Count 2020-08-01 Aggression 0.09 2020-08-01 Elopement 0.00 2020-08-01 Self-injury 0.97 2020-09-01 Aggression 0.65 2020-09-01 Elopement 0.00 2020-09-01 Self-injury 1.58 2020-10-01 Aggression 0.24 2020-10-01 Elopement 0.00 2020-10-01 Self-injury 0.75 2020-11-01 Aggression 0.03 2020-11-01 Elopement 0.01 2020-11-01 Self-injury 0.89 2020-12-01 Aggression 0.14 2020-12-01 Elopement 0.00 2020-12-01 Self-injury 0.94 2021-01-01 Aggression 0.05 2021-01-01 Elopement 0.00 2021-01-01 Self-injury 0.30 2021-02-01 Self-injury 0.42 2021-02-01 Elopement 0.03 2021-02-01 Aggression 0.16 2021-03-01 Elopement 0.00 2021-03-01 Self-injury 0.68 2021-03-01 Aggression 0.20 2021-04-01 Aggression 0.10 2021-04-01 Elopement 0.03 2021-04-01 Self-injury 0.33 2021-05-01 Elopement 0.20 2021-05-01 Aggression 0.21 2021-05-01 Self-injury 1.63 2021-06-01 Self-injury 0.90 2021-06-01 Aggression 0.29 2021-06-01 Elopement 0.14  I find this strange as I'm directly following the documentation - https://plotly.com/python/linear-fits/ Is this a known issue? I can't find any examples with a google search...

Answer: As Henry helpfully pointed out this was just a version problem, easily addressed by updating plotly using: pip install plotly==5.2.1
Question: I have a question about appending a dataframe to existing sheet on existing file. I tried to write the code by myself writer = pd.ExcelWriter('existingFile.xlsx', engine='openpyxl', mode='a') df.to_excel(writer, sheet_name="existingSheet", startrow=writer.sheets["existingSheet"].max_row, index=False, header=False)  and this would cause an error ValueError: Sheet 'existingSheet' already exists and if_sheet_exists is set to 'error'. and I googled and found this function in here; Append existing excel sheet with new dataframe using python pandas and even with this function, it still causes the same error, even though i think this function prevents this exact error from what i think. Could you please help? Thank you very much!

Answer: It seems this function is broken in pandas 1.3.0 Just look at this document , when trying to append to existing sheet it will either raise an error, create a new one or replace it  if_sheet_exists{‘error’, ‘new’, ‘replace’}, default ‘error’ How to behave when trying to write to a sheet that already exists (append mode only). error: raise a ValueError. new: Create a new sheet, with a name determined by the engine. replace: Delete the contents of the sheet before writing to it. New in version 1.3.0  The only solution is to downgrade pandas (1.2.3 is working for me) pip install pandas==1.2.3
Question: I have a GeoDataFrame (let's call it Destinations) that was made from a point shapefile using GeoPandas. For every feature (correct me if the terminology is wrong) in Destinations, I need to find the nearest node on a graph and save that node to another GeoDataFrame (let's call it Destination_nodes, it will be used later for shortest-path analysis). Presumably, this requires creating a new, blank Destination_nodes GeoDataFrame and appending nodes to it as I loop through Destinations. But what is the syntax for creating a new empty GeoDataFrame?

Answer: For a GeoDataFrame you need to say which column will be the geometry, i.e. contain features as you say. So you could simply specify the columns of your empty dataframe at creation, without specifying any data: >>> dest = geopandas.GeoDataFrame(columns=['id', 'distance', 'feature'], geometry='feature') >>> dest Empty GeoDataFrame Columns: [id, distance, feature] Index: [] >>> dest.geometry GeoSeries([], Name: feature, dtype: geometry)
Question: I assumed that spark.sql(query) is used when we are using spark sql and that spark.read.format("jdbc").option("query", "") is used when we are using oracle sql syntax. Would I be right in assuming so?

Answer: Yes and Spark does more than that too! Spark-Jdbc: From Spark docs Jdbc(Java Database connectivity) is used to read/write data from other databases (oracle, mysql, sqlserver, postgres, db2..etc). spark.read.format("jdbc").option("query", "(select * from <db>.<tb>)e") Spark-Sql: From docs Spark's module for working with structured data and lets you to query data using DataFrame API or SQL API.  We can use Spark-Sql to read data from hive/presto/athena/delta/csv/parquet...etc files. Create a temporary view/table on dataframe then run SQL queries. Easily write RDD's/Dataframes to Hive or Parquet files.
Question: I'm new to pandas and decided to learn it by playing around with some data I pulled from my favorite game's API. I have a dataframe with two columns "playerId" and "winner" like so: playerStatus: ______________________ playerId winner 0 1848 True 1 1988 False 2 3543 True 3 1848 False 4 1988 False ...  Each row represents a match the player participated in. My goal is to either transform this dataframe or create a new one such that the win percentage for each playerId is calculated. For example, the above dataframe would become: playerWinsAndTotals _________________________________________ playerId wins totalPlayed winPct 0 1848 1 2 50.0000 1 1988 0 2 0.0000 2 3543 1 1 100.0000 ...  It took quite a while of reading pandas docs, but I actually managed to achieve this by essentially creating two different tables (one to find the number of wins for each player, one to find the total games for each player), and merging them, then taking the ratio of wins to games played. Creating the "wins" dataframe: temp_df = playerStatus[['playerId', 'winner']].value_counts().reset_index(name='wins') onlyWins = temp_df[temp_df['winner'] == True][['playerId', 'wins']] onlyWins _________________________ playerId wins 1 1670 483 3 1748 474 4 2179 468 6 4006 434 8 1668 392 ...  Creating the "totals" dataframe: totalPlayed = playerStatus['playerId'].value_counts().reset_index(name='totalCount').rename(columns={'index': 'playerId'}) totalPlayed ____________________ playerId totalCount 0 1670 961 1 1748 919 2 1872 877 3 4006 839 4 2179 837 ...  Finally, merging them and adding the "winPct" column. playerWinsAndTotals = onlyWins.merge(totalPlayed, on='playerId', how='left') playerWinsAndTotals['winPct'] = playerWinsAndTotals['wins']/playerWinsAndTotals['totalCount'] * 100 playerWinsAndTotals _____________________________________________ playerId wins totalCount winPct 0 1670 483 961 50.260146 1 1748 474 919 51.577802 2 2179 468 837 55.913978 3 4006 434 839 51.728248 4 1668 392 712 55.056180 ...  Now, the reason I am posting this here is because I know I'm not taking full advantage of what pandas has to offer. Creating and merging two different dataframes just to find the ratio of player wins seems unnecessary. I feel like I took the "scenic" route on this one. To anyone more experienced than me, how would you tackle this problem?

Answer: We can take advantage of the way that Boolean values are handled mathematically (True being 1 and False being 0) and use 3 aggregation functions sum, count and mean per group (groupby aggregate). We can also take advantage of Named Aggregation to both create and rename the columns in one step: df = ( df.groupby('playerId', as_index=False) .agg(wins=('winner', 'sum'), totalCount=('winner', 'count'), winPct=('winner', 'mean')) ) # Scale up winPct df['winPct'] *= 100  df:  playerId wins totalCount winPct 0 1848 1 2 50.0 1 1988 0 2 0.0 2 3543 1 1 100.0   DataFrame and imports: import pandas as pd df = pd.DataFrame({ 'playerId': [1848, 1988, 3543, 1848, 1988], 'winner': [True, False, True, False, False] })
Question: I have a multi=indexed DataFrame, but I want to keep only two columns per level 1, for each of the level 0 variables (i.e. columns 'one' and 'two'). I can subset them separately, but I would like to do it together so I can keep the values side by side Here is the DataFrame index = pd.MultiIndex.from_tuples(list(zip(*[['bar1', 'foo1', 'bar1', 'foo2','bar3','foo3'], ['one','two','three','two','one','four']]))) df = pd.DataFrame(np.random.randn(2, 6), columns=index)  Here is the way to subset for one column in level 1 df.iloc[:, df.columns.get_level_values(1)== 'one'] # or df.xs('one', level=1, axis=1) # but adding two columns within either command will not work e.g. df.xs(('one','two), level=1, axis=1)  This would be the expected output  bar1 foo1 foo2 bar3 one two two one 0 -0.508272 -0.195379 0.865563 2.002205 1 -0.771565 1.360479 1.900931 -1.589277 

Answer: Here's one way using pd.IndexSlice: idnx = pd.IndexSlice[:, ['one', 'two']] df.loc[:, idnx]  Output:  bar1 bar3 foo1 foo2 one one two two 0 0.589999 0.261224 -0.106588 -2.309628 1 0.646201 -0.491110 0.430724 1.027424   Another way using a little known argument, axis, of pd.DataFrame.loc: df.loc(axis=1)[:, ['one', 'two']]  Output:  bar1 bar3 foo1 foo2 one one two two 0 0.589999 0.261224 -0.106588 -2.309628 1 0.646201 -0.491110 0.430724 1.027424  NOTE: This argument is not listed in the documented API for pd.DataFrame.loc, but is referenced in the User Guide in MultiIndex / Advanced indexing section in the Using Slicers paragraph about middle way down with an example.
Question: I am new to python and am trying to run a python 2.7 script. Got pip for python 2.7 and installed a dependency of pyCrypto from the mac terminal shell. The downloaded python script, I want to try, runs fine in the terminal app when I execute it using python2. Now I open it in vscode and try to run the script in its terminal and I get ImportError: dlopen(/Users/xxx/Library/Python/2.7/lib/python/site-packages/Crypto/Cipher/_DES3.so, 2): no suitable image found. Did find: /Users/xxx/Library/Python/2.7/lib/python/site-packages/Crypto/Cipher/_DES3.so: mach-o, but wrong architecture /Users/xxx/Library/Python/2.7/lib/python/site-packages/Crypto/Cipher/_DES3.so: mach-o, but wrong architecture  When I run uname -m inside a vscode terminal(zsh) on an M1 Mac, I see an output of x86_64, implying the terminal is running under Rosetta and looking for the intel version of the library. And when I run uname -m in the regular mac terminal app, I see arm64 How do I change the terminal inside vscode to use the non rosetta one? Or how do I get the script to run from within vscode?

Answer: I'm not familiar with VSCode, but you can manually force the chosen architecture slice of anything you launch with the arch command (see man arch). If you have a script that you'd normally launch like: ./script.py  Then you can force either architecture like so: arch -x86_64 ./script.py arch -arm64 ./script.py
Question: Please see the below code snippet: In [1]: class A(Exception): ...: def __init__(self, b): ...: self.message = b.message ...: In [2]: class B: ...: message = "hello" ...: In [3]: A(B()) Out[3]: __main__.A(<__main__.B at 0x14af96790>) In [4]: class A: ...: def __init__(self, b): ...: self.message = b.message ...: In [5]: A(B()) Out[5]: <__main__.A at 0x10445b0a0>  if A subclasses from Exception, its repr returns a reference to B() even though we only pass B()'s message attribute. Why is this the intentional behavior in Exception.repr and how does it work in python psuedocode if possible given the cpython code isn't too readable ?

Answer: When a Python object is created, it is the class's __new__ method that is called, and __init__ is then called on the new instance that the __new__ method returns (assuming it returned a new instance, which sometimes it doesn't). Your overridden __init__ method doesn't keep a reference to b, but you didn't override __new__, so you inherit the __new__ method defined here (CPython source link): static PyObject * BaseException_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { // ... if (args) { self->args = args; Py_INCREF(args); return (PyObject *)self; } // ... }  I omitted the parts that are not relevant. As you can see, the __new__ method of the BaseException class stores a reference to the tuple of arguments used when creating the exception, and this tuple is therefore available for the __repr__ method to print a reference to the objects used to instantiate the exception. So it's this tuple which retains a reference to the original argument b. This is consistent with the general expectation that repr should return Python code which would create an object in the same state, if it can. Note that it is only args, not kwds, which has this behaviour; the __new__ method doesn't store a reference to kwds and __repr__ doesn't print it, so we should expect not to see the same behaviour if the constructor is called with a keyword argument instead of a positional argument. Indeed, that is what we observe: >>> A(B()) A(<__main__.B object at 0x7fa8e7a23860>,) >>> A(b=B()) A()  A bit strange since the two A objects are supposed to have the same state, but that's how the code is written, anyway.