neulab/docprompting-conala huggingface dataset:
pandas.DataFrame.to_parquet   DataFrame.to_parquet(path=None, engine='auto', compression='snappy', index=None, partition_cols=None, storage_options=None, **kwargs)[source]
 
Write a DataFrame to the binary parquet format. This function writes the dataframe as a parquet file. You can choose different parquet backends, and have the option of compression. See the user guide for more details.  Parameters 
 
path:str, path object, file-like object, or None, default None


String, path object (implementing os.PathLike[str]), or file-like object implementing a binary write() function. If None, the result is returned as bytes. If a string or path, it will be used as Root Directory path when writing a partitioned dataset.  Changed in version 1.2.0.  Previously this was “fname”  
engine:{‘auto’, ‘pyarrow’, ‘fastparquet’}, default ‘auto’


Parquet library to use. If ‘auto’, then the option io.parquet.engine is used. The default io.parquet.engine behavior is to try ‘pyarrow’, falling back to ‘fastparquet’ if ‘pyarrow’ is unavailable.  
compression:{‘snappy’, ‘gzip’, ‘brotli’, None}, default ‘snappy’


Name of the compression to use. Use None for no compression.  
index:bool, default None


If True, include the dataframe’s index(es) in the file output. If False, they will not be written to the file. If None, similar to True the dataframe’s index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn’t require much space and is faster. Other indexes will be included as columns in the file output.  
partition_cols:list, optional, default None


Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string.  
storage_options:dict, optional


Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to urllib as header options. For other URLs (e.g. starting with “s3://”, and “gcs://”) the key-value pairs are forwarded to fsspec. Please see fsspec and urllib for more details.  New in version 1.2.0.   **kwargs

Additional arguments passed to the parquet library. See pandas io for more details.    Returns 
 bytes if no path argument is provided else None
    See also  read_parquet

Read a parquet file.  DataFrame.to_csv

Write a csv file.  DataFrame.to_sql

Write to a sql table.  DataFrame.to_hdf

Write to hdf.    Notes This function requires either the fastparquet or pyarrow library. Examples 
>>> df = pd.DataFrame(data={'col1': [1, 2], 'col2': [3, 4]})
>>> df.to_parquet('df.parquet.gzip',
...               compression='gzip')  
>>> pd.read_parquet('df.parquet.gzip')  
   col1  col2
0     1     3
1     2     4
  If you want to get a buffer to the parquet content you can use a io.BytesIO object, as long as you don’t use partition_cols, which creates multiple files. 
>>> import io
>>> f = io.BytesIO()
>>> df.to_parquet(f)
>>> f.seek(0)
0
>>> content = f.read()
pandas.DataFrame.to_period   DataFrame.to_period(freq=None, axis=0, copy=True)[source]
 
Convert DataFrame from DatetimeIndex to PeriodIndex. Convert DataFrame from DatetimeIndex to PeriodIndex with desired frequency (inferred from index if not passed).  Parameters 
 
freq:str, default


Frequency of the PeriodIndex.  
axis:{0 or ‘index’, 1 or ‘columns’}, default 0


The axis to convert (the index by default).  
copy:bool, default True


If False then underlying input data is not copied.    Returns 
 DataFrame with PeriodIndex
   Examples 
>>> idx = pd.to_datetime(
...     [
...         "2001-03-31 00:00:00",
...         "2002-05-31 00:00:00",
...         "2003-08-31 00:00:00",
...     ]
... )
  
>>> idx
DatetimeIndex(['2001-03-31', '2002-05-31', '2003-08-31'],
dtype='datetime64[ns]', freq=None)
  
>>> idx.to_period("M")
PeriodIndex(['2001-03', '2002-05', '2003-08'], dtype='period[M]')
  For the yearly frequency 
>>> idx.to_period("Y")
PeriodIndex(['2001', '2002', '2003'], dtype='period[A-DEC]')
pandas.DataFrame.to_pickle   DataFrame.to_pickle(path, compression='infer', protocol=5, storage_options=None)[source]
 
Pickle (serialize) object to file.  Parameters 
 
path:str


File path where the pickled object will be stored.  
compression:str or dict, default ‘infer’


For on-the-fly compression of the output data. If ‘infer’ and ‘path’ path-like, then detect compression from the following extensions: ‘.gz’, ‘.bz2’, ‘.zip’, ‘.xz’, or ‘.zst’ (otherwise no compression). Set to None for no compression. Can also be a dict with key 'method' set to one of {'zip', 'gzip', 'bz2', 'zstd'} and other key-value pairs are forwarded to zipfile.ZipFile, gzip.GzipFile, bz2.BZ2File, or zstandard.ZstdDecompressor, respectively. As an example, the following could be passed for faster compression and to create a reproducible gzip archive: compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1}.  
protocol:int


Int which indicates which protocol should be used by the pickler, default HIGHEST_PROTOCOL (see [1] paragraph 12.1.2). The possible values are 0, 1, 2, 3, 4, 5. A negative value for the protocol parameter is equivalent to setting its value to HIGHEST_PROTOCOL.  1 
https://docs.python.org/3/library/pickle.html.    
storage_options:dict, optional


Extra options that make sense for a particular storage connection, e.g. host, port, username, password, etc. For HTTP(S) URLs the key-value pairs are forwarded to urllib as header options. For other URLs (e.g. starting with “s3://”, and “gcs://”) the key-value pairs are forwarded to fsspec. Please see fsspec and urllib for more details.  New in version 1.2.0.       See also  read_pickle

Load pickled pandas object (or any object) from file.  DataFrame.to_hdf

Write DataFrame to an HDF5 file.  DataFrame.to_sql

Write DataFrame to a SQL database.  DataFrame.to_parquet

Write a DataFrame to the binary parquet format.    Examples 
>>> original_df = pd.DataFrame({"foo": range(5), "bar": range(5, 10)})  
>>> original_df  
   foo  bar
0    0    5
1    1    6
2    2    7
3    3    8
4    4    9
>>> original_df.to_pickle("./dummy.pkl")  
  
>>> unpickled_df = pd.read_pickle("./dummy.pkl")  
>>> unpickled_df  
   foo  bar
0    0    5
1    1    6
2    2    7
3    3    8
4    4    9
pandas.DataFrame.to_records   DataFrame.to_records(index=True, column_dtypes=None, index_dtypes=None)[source]
 
Convert DataFrame to a NumPy record array. Index will be included as the first field of the record array if requested.  Parameters 
 
index:bool, default True


Include index in resulting record array, stored in ‘index’ field or using the index label, if set.  
column_dtypes:str, type, dict, default None


If a string or type, the data type to store all columns. If a dictionary, a mapping of column names and indices (zero-indexed) to specific data types.  
index_dtypes:str, type, dict, default None


If a string or type, the data type to store all index levels. If a dictionary, a mapping of index level names and indices (zero-indexed) to specific data types. This mapping is applied only if index=True.    Returns 
 numpy.recarray

NumPy ndarray with the DataFrame labels as fields and each row of the DataFrame as entries.      See also  DataFrame.from_records

Convert structured or record ndarray to DataFrame.  numpy.recarray

An ndarray that allows field access using attributes, analogous to typed columns in a spreadsheet.    Examples 
>>> df = pd.DataFrame({'A': [1, 2], 'B': [0.5, 0.75]},
...                   index=['a', 'b'])
>>> df
   A     B
a  1  0.50
b  2  0.75
>>> df.to_records()
rec.array([('a', 1, 0.5 ), ('b', 2, 0.75)],
          dtype=[('index', 'O'), ('A', '<i8'), ('B', '<f8')])
  If the DataFrame index has no label then the recarray field name is set to ‘index’. If the index has a label then this is used as the field name: 
>>> df.index = df.index.rename("I")
>>> df.to_records()
rec.array([('a', 1, 0.5 ), ('b', 2, 0.75)],
          dtype=[('I', 'O'), ('A', '<i8'), ('B', '<f8')])
  The index can be excluded from the record array: 
>>> df.to_records(index=False)
rec.array([(1, 0.5 ), (2, 0.75)],
          dtype=[('A', '<i8'), ('B', '<f8')])
  Data types can be specified for the columns: 
>>> df.to_records(column_dtypes={"A": "int32"})
rec.array([('a', 1, 0.5 ), ('b', 2, 0.75)],
          dtype=[('I', 'O'), ('A', '<i4'), ('B', '<f8')])
  As well as for the index: 
>>> df.to_records(index_dtypes="<S2")
rec.array([(b'a', 1, 0.5 ), (b'b', 2, 0.75)],
          dtype=[('I', 'S2'), ('A', '<i8'), ('B', '<f8')])
  
>>> index_dtypes = f"<S{df.index.str.len().max()}"
>>> df.to_records(index_dtypes=index_dtypes)
rec.array([(b'a', 1, 0.5 ), (b'b', 2, 0.75)],
          dtype=[('I', 'S1'), ('A', '<i8'), ('B', '<f8')])
pandas.DataFrame.to_timestamp   DataFrame.to_timestamp(freq=None, how='start', axis=0, copy=True)[source]
 
Cast to DatetimeIndex of timestamps, at beginning of period.  Parameters 
 
freq:str, default frequency of PeriodIndex


Desired frequency.  
how:{‘s’, ‘e’, ‘start’, ‘end’}


Convention for converting period to timestamp; start of period vs. end.  
axis:{0 or ‘index’, 1 or ‘columns’}, default 0


The axis to convert (the index by default).  
copy:bool, default True


If False then underlying input data is not copied.    Returns 
 DataFrame with DatetimeIndex
pandas.DataFrame.to_xarray   DataFrame.to_xarray()[source]
 
Return an xarray object from the pandas object.  Returns 
 xarray.DataArray or xarray.Dataset

Data in the pandas structure converted to Dataset if the object is a DataFrame, or a DataArray if the object is a Series.      See also  DataFrame.to_hdf

Write DataFrame to an HDF5 file.  DataFrame.to_parquet

Write a DataFrame to the binary parquet format.    Notes See the xarray docs Examples 
>>> df = pd.DataFrame([('falcon', 'bird', 389.0, 2),
...                    ('parrot', 'bird', 24.0, 2),
...                    ('lion', 'mammal', 80.5, 4),
...                    ('monkey', 'mammal', np.nan, 4)],
...                   columns=['name', 'class', 'max_speed',
...                            'num_legs'])
>>> df
     name   class  max_speed  num_legs
0  falcon    bird      389.0         2
1  parrot    bird       24.0         2
2    lion  mammal       80.5         4
3  monkey  mammal        NaN         4
  
>>> df.to_xarray()
<xarray.Dataset>
Dimensions:    (index: 4)
Coordinates:
  * index      (index) int64 0 1 2 3
Data variables:
    name       (index) object 'falcon' 'parrot' 'lion' 'monkey'
    class      (index) object 'bird' 'bird' 'mammal' 'mammal'
    max_speed  (index) float64 389.0 24.0 80.5 nan
    num_legs   (index) int64 2 2 4 4
  
>>> df['max_speed'].to_xarray()
<xarray.DataArray 'max_speed' (index: 4)>
array([389. ,  24. ,  80.5,   nan])
Coordinates:
  * index    (index) int64 0 1 2 3
  
>>> dates = pd.to_datetime(['2018-01-01', '2018-01-01',
...                         '2018-01-02', '2018-01-02'])
>>> df_multiindex = pd.DataFrame({'date': dates,
...                               'animal': ['falcon', 'parrot',
...                                          'falcon', 'parrot'],
...                               'speed': [350, 18, 361, 15]})
>>> df_multiindex = df_multiindex.set_index(['date', 'animal'])
  
>>> df_multiindex
                   speed
date       animal
2018-01-01 falcon    350
           parrot     18
2018-01-02 falcon    361
           parrot     15
  
>>> df_multiindex.to_xarray()
<xarray.Dataset>
Dimensions:  (animal: 2, date: 2)
Coordinates:
  * date     (date) datetime64[ns] 2018-01-01 2018-01-02
  * animal   (animal) object 'falcon' 'parrot'
Data variables:
    speed    (date, animal) int64 350 18 361 15
pandas.DataFrame.transform   DataFrame.transform(func, axis=0, *args, **kwargs)[source]
 
Call func on self producing a DataFrame with the same axis shape as self.  Parameters 
 
func:function, str, list-like or dict-like


Function to use for transforming the data. If a function, must either work when passed a DataFrame or when passed to DataFrame.apply. If func is both list-like and dict-like, dict-like behavior takes precedence. Accepted combinations are:  function string function name list-like of functions and/or function names, e.g. [np.exp, 'sqrt'] dict-like of axis labels -> functions, function names or list-like of such.   
axis:{0 or ‘index’, 1 or ‘columns’}, default 0


If 0 or ‘index’: apply function to each column. If 1 or ‘columns’: apply function to each row.  *args

Positional arguments to pass to func.  **kwargs

Keyword arguments to pass to func.    Returns 
 DataFrame

A DataFrame that must have the same length as self.    Raises 
 
ValueError:If the returned DataFrame has a different length than self.

    See also  DataFrame.agg

Only perform aggregating type operations.  DataFrame.apply

Invoke function on a DataFrame.    Notes Functions that mutate the passed object can produce unexpected behavior or errors and are not supported. See Mutating with User Defined Function (UDF) methods for more details. Examples 
>>> df = pd.DataFrame({'A': range(3), 'B': range(1, 4)})
>>> df
   A  B
0  0  1
1  1  2
2  2  3
>>> df.transform(lambda x: x + 1)
   A  B
0  1  2
1  2  3
2  3  4
  Even though the resulting DataFrame must have the same length as the input DataFrame, it is possible to provide several input functions: 
>>> s = pd.Series(range(3))
>>> s
0    0
1    1
2    2
dtype: int64
>>> s.transform([np.sqrt, np.exp])
       sqrt        exp
0  0.000000   1.000000
1  1.000000   2.718282
2  1.414214   7.389056
  You can call transform on a GroupBy object: 
>>> df = pd.DataFrame({
...     "Date": [
...         "2015-05-08", "2015-05-07", "2015-05-06", "2015-05-05",
...         "2015-05-08", "2015-05-07", "2015-05-06", "2015-05-05"],
...     "Data": [5, 8, 6, 1, 50, 100, 60, 120],
... })
>>> df
         Date  Data
0  2015-05-08     5
1  2015-05-07     8
2  2015-05-06     6
3  2015-05-05     1
4  2015-05-08    50
5  2015-05-07   100
6  2015-05-06    60
7  2015-05-05   120
>>> df.groupby('Date')['Data'].transform('sum')
0     55
1    108
2     66
3    121
4     55
5    108
6     66
7    121
Name: Data, dtype: int64
  
>>> df = pd.DataFrame({
...     "c": [1, 1, 1, 2, 2, 2, 2],
...     "type": ["m", "n", "o", "m", "m", "n", "n"]
... })
>>> df
   c type
0  1    m
1  1    n
2  1    o
3  2    m
4  2    m
5  2    n
6  2    n
>>> df['size'] = df.groupby('c')['type'].transform(len)
>>> df
   c type size
0  1    m    3
1  1    n    3
2  1    o    3
3  2    m    4
4  2    m    4
5  2    n    4
6  2    n    4
pandas.DataFrame.transpose   DataFrame.transpose(*args, copy=False)[source]
 
Transpose index and columns. Reflect the DataFrame over its main diagonal by writing rows as columns and vice-versa. The property T is an accessor to the method transpose().  Parameters 
 
*args:tuple, optional


Accepted for compatibility with NumPy.  
copy:bool, default False


Whether to copy the data after transposing, even for DataFrames with a single dtype. Note that a copy is always required for mixed dtype DataFrames, or for DataFrames with any extension types.    Returns 
 DataFrame

The transposed DataFrame.      See also  numpy.transpose

Permute the dimensions of a given array.    Notes Transposing a DataFrame with mixed dtypes will result in a homogeneous DataFrame with the object dtype. In such a case, a copy of the data is always made. Examples Square DataFrame with homogeneous dtype 
>>> d1 = {'col1': [1, 2], 'col2': [3, 4]}
>>> df1 = pd.DataFrame(data=d1)
>>> df1
   col1  col2
0     1     3
1     2     4
  
>>> df1_transposed = df1.T # or df1.transpose()
>>> df1_transposed
      0  1
col1  1  2
col2  3  4
  When the dtype is homogeneous in the original DataFrame, we get a transposed DataFrame with the same dtype: 
>>> df1.dtypes
col1    int64
col2    int64
dtype: object
>>> df1_transposed.dtypes
0    int64
1    int64
dtype: object
  Non-square DataFrame with mixed dtypes 
>>> d2 = {'name': ['Alice', 'Bob'],
...       'score': [9.5, 8],
...       'employed': [False, True],
...       'kids': [0, 0]}
>>> df2 = pd.DataFrame(data=d2)
>>> df2
    name  score  employed  kids
0  Alice    9.5     False     0
1    Bob    8.0      True     0
  
>>> df2_transposed = df2.T # or df2.transpose()
>>> df2_transposed
              0     1
name      Alice   Bob
score       9.5   8.0
employed  False  True
kids          0     0
  When the DataFrame has mixed dtypes, we get a transposed DataFrame with the object dtype: 
>>> df2.dtypes
name         object
score       float64
employed       bool
kids          int64
dtype: object
>>> df2_transposed.dtypes
0    object
1    object
dtype: object
pandas.DataFrame.truncate   DataFrame.truncate(before=None, after=None, axis=None, copy=True)[source]
 
Truncate a Series or DataFrame before and after some index value. This is a useful shorthand for boolean indexing based on index values above or below certain thresholds.  Parameters 
 
before:date, str, int


Truncate all rows before this index value.  
after:date, str, int


Truncate all rows after this index value.  
axis:{0 or ‘index’, 1 or ‘columns’}, optional


Axis to truncate. Truncates the index (rows) by default.  
copy:bool, default is True,


Return a copy of the truncated section.    Returns 
 type of caller

The truncated Series or DataFrame.      See also  DataFrame.loc

Select a subset of a DataFrame by label.  DataFrame.iloc

Select a subset of a DataFrame by position.    Notes If the index being truncated contains only datetime values, before and after may be specified as strings instead of Timestamps. Examples 
>>> df = pd.DataFrame({'A': ['a', 'b', 'c', 'd', 'e'],
...                    'B': ['f', 'g', 'h', 'i', 'j'],
...                    'C': ['k', 'l', 'm', 'n', 'o']},
...                   index=[1, 2, 3, 4, 5])
>>> df
   A  B  C
1  a  f  k
2  b  g  l
3  c  h  m
4  d  i  n
5  e  j  o
  
>>> df.truncate(before=2, after=4)
   A  B  C
2  b  g  l
3  c  h  m
4  d  i  n
  The columns of a DataFrame can be truncated. 
>>> df.truncate(before="A", after="B", axis="columns")
   A  B
1  a  f
2  b  g
3  c  h
4  d  i
5  e  j
  For Series, only rows can be truncated. 
>>> df['A'].truncate(before=2, after=4)
2    b
3    c
4    d
Name: A, dtype: object
  The index values in truncate can be datetimes or string dates. 
>>> dates = pd.date_range('2016-01-01', '2016-02-01', freq='s')
>>> df = pd.DataFrame(index=dates, data={'A': 1})
>>> df.tail()
                     A
2016-01-31 23:59:56  1
2016-01-31 23:59:57  1
2016-01-31 23:59:58  1
2016-01-31 23:59:59  1
2016-02-01 00:00:00  1
  
>>> df.truncate(before=pd.Timestamp('2016-01-05'),
...             after=pd.Timestamp('2016-01-10')).tail()
                     A
2016-01-09 23:59:56  1
2016-01-09 23:59:57  1
2016-01-09 23:59:58  1
2016-01-09 23:59:59  1
2016-01-10 00:00:00  1
  Because the index is a DatetimeIndex containing only dates, we can specify before and after as strings. They will be coerced to Timestamps before truncation. 
>>> df.truncate('2016-01-05', '2016-01-10').tail()
                     A
2016-01-09 23:59:56  1
2016-01-09 23:59:57  1
2016-01-09 23:59:58  1
2016-01-09 23:59:59  1
2016-01-10 00:00:00  1
  Note that truncate assumes a 0 value for any unspecified time component (midnight). This differs from partial string slicing, which returns any partially matching dates. 
>>> df.loc['2016-01-05':'2016-01-10', :].tail()
                     A
2016-01-10 23:59:55  1
2016-01-10 23:59:56  1
2016-01-10 23:59:57  1
2016-01-10 23:59:58  1
2016-01-10 23:59:59  1
pandas.DataFrame.tshift   DataFrame.tshift(periods=1, freq=None, axis=0)[source]
 
Shift the time index, using the index’s frequency if available.  Deprecated since version 1.1.0: Use shift instead.   Parameters 
 
periods:int


Number of periods to move, can be positive or negative.  
freq:DateOffset, timedelta, or str, default None


Increment to use from the tseries module or time rule expressed as a string (e.g. ‘EOM’).  
axis:{0 or ‘index’, 1 or ‘columns’, None}, default 0


Corresponds to the axis that contains the Index.    Returns 
 
shifted:Series/DataFrame

   Notes If freq is not specified then tries to use the freq or inferred_freq attributes of the index. If neither of those attributes exist, a ValueError is thrown
pandas.DataFrame.tz_convert   DataFrame.tz_convert(tz, axis=0, level=None, copy=True)[source]
 
Convert tz-aware axis to target time zone.  Parameters 
 
tz:str or tzinfo object


axis:the axis to convert


level:int, str, default None


If axis is a MultiIndex, convert a specific level. Otherwise must be None.  
copy:bool, default True


Also make a copy of the underlying data.    Returns 
 {klass}

Object with time zone converted axis.    Raises 
 TypeError

If the axis is tz-naive.
pandas.DataFrame.unstack   DataFrame.unstack(level=- 1, fill_value=None)[source]
 
Pivot a level of the (necessarily hierarchical) index labels. Returns a DataFrame having a new level of column labels whose inner-most level consists of the pivoted index labels. If the index is not a MultiIndex, the output will be a Series (the analogue of stack when the columns are not a MultiIndex).  Parameters 
 
level:int, str, or list of these, default -1 (last level)


Level(s) of index to unstack, can pass level name.  
fill_value:int, str or dict


Replace NaN with this value if the unstack produces missing values.    Returns 
 Series or DataFrame
    See also  DataFrame.pivot

Pivot a table based on column values.  DataFrame.stack

Pivot a level of the column labels (inverse operation from unstack).    Examples 
>>> index = pd.MultiIndex.from_tuples([('one', 'a'), ('one', 'b'),
...                                    ('two', 'a'), ('two', 'b')])
>>> s = pd.Series(np.arange(1.0, 5.0), index=index)
>>> s
one  a   1.0
     b   2.0
two  a   3.0
     b   4.0
dtype: float64
  
>>> s.unstack(level=-1)
     a   b
one  1.0  2.0
two  3.0  4.0
  
>>> s.unstack(level=0)
   one  two
a  1.0   3.0
b  2.0   4.0
  
>>> df = s.unstack(level=0)
>>> df.unstack()
one  a  1.0
     b  2.0
two  a  3.0
     b  4.0
dtype: float64