Package: xv
xvAccess to arxiv dataTo install:pip install xvExamplesfromxvimport*Raw storeAt the point of writing this, my attempts enablegrazeto automatically confirm download in the googledrive downloads (which, when downloading too-big files, will tell the user it can't scan the file and ask the user to confirm the download).Therefore, the following files need to be downloaded manually:titles:https://drive.google.com/file/d/1Ul5mPePtoPKHZkH5Rm6dWKAO11dG98GN/view?usp=share_linkabstracts:https://drive.google.com/file/d/1g3K-wlixFxklTSUQNZKpEgN4WNTFTPIZ/view?usp=share_link(If those urls don't work, perhaps they were updated: See here:https://alex.macrocosm.so/download.)You can then copy them over to the place graze will look for by doing:frompathlibimportPathfromxv.utilimportGrazefromxv.data_accessimporturlsg[urls['titles']]=Path('TITLES_DATA_LOCAL_FILEPATH').read_bytes()g[urls['abstracts']]=Path('ABSTRACTS_DATA_LOCAL_FILEPATH').read_bytes()# from imbed.mdat.arxiv import urls# from pathlib import Path# g[urls['titles']] = Path('FILE_WHERE_YOU_DOWNLOADED_TITLES_DATA').read_bytes()# g[urls['abstracts']] = Path('FILE_WHERE_YOU_DOWNLOADED_TITLES_DATA').read_bytes()fromxv.utilimportGrazeg=Graze()list(g)['https://drive.google.com/file/d/1Ul5mPePtoPKHZkH5Rm6dWKAO11dG98GN/view?usp=share_link',
 'https://drive.google.com/file/d/1g3K-wlixFxklTSUQNZKpEgN4WNTFTPIZ/view?usp=share_link',
 'https://arxiv.org/pdf/0704.0001']fromxvimportraw_sourceslist(raw_sources)['titles', 'abstracts']raw=raw_sources['titles']list(raw)['titles_7.parquet',
 'titles_23.parquet',
 'titles_15.parquet',
 'verifyResults.py',
 'titles_14.parquet',
 'titles_22.parquet',
 'titles_6.parquet',
 'titles_16.parquet',
 'titles_20.parquet',
 'titles_4.parquet',
 'titles_5.parquet',
 'titles_21.parquet',
 'params.txt',
 'titles_17.parquet',
 'exampleEmbed.py',
 'titles_12.parquet',
 'README.md',
 'titles_9.parquet',
 'titles_1.parquet',
 'titles_13.parquet',
 'titles_8.parquet',
 'titles_18.parquet',
 'titles_3.parquet',
 'titles_11.parquet',
 'titles_10.parquet',
 'titles_19.parquet',
 'titles_2.parquet']print(raw['exampleEmbed.py'].decode())from InstructorEmbedding import INSTRUCTOR

model = INSTRUCTOR('hkunlp/instructor-xl')
sentence = "3D ActionSLAM: wearable person tracking in multi-floor environments"
instruction = "Represent the Research Paper title for retrieval; Input:"
embeddings = model.encode([[instruction,sentence]])
print(embeddings)fromInstructorEmbeddingimportINSTRUCTORmodel=INSTRUCTOR('hkunlp/instructor-xl')/Users/thorwhalen/.pyenv/versions/3.10.13/envs/p10/lib/python3.10/site-packages/InstructorEmbedding/instructor.py:7: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  from tqdm.autonotebook import trange


load INSTRUCTOR_Transformer
max_seq_length  512sentence="3D ActionSLAM: wearable person tracking in multi-floor environments"instruction="Represent the Research Paper title for retrieval; Input:"embeddings=model.encode([[instruction,sentence]])print(raw['params.txt'].decode())prompt: Represent the Research Paper title for retrieval; Input:
type: title
time string: 20230518-185428
model: InstructorXL
version: 2.0print(raw['exampleEmbed.py'].decode())from InstructorEmbedding import INSTRUCTOR

model = INSTRUCTOR('hkunlp/instructor-xl')
sentence = "3D ActionSLAM: wearable person tracking in multi-floor environments"
instruction = "Represent the Research Paper title for retrieval; Input:"
embeddings = model.encode([[instruction,sentence]])
print(embeddings)The imbedding data storeAnd now, we'll transform the raw store to get a convenient interface to the actual data of interest.b=raw['titles_1.parquet']len(b)313383694fromxvimportsources# raw store + wrapper. See parquet_codec code.titles_tables=sources['titles']abstract_tables=sources['abstracts']print(list(titles_tables))[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]titles_df=titles_tables[1]titles_df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>titleembeddingsdoi0Calculation of prompt diphoton production cros...[-0.050620172, 0.041436385, 0.05363288, -0.029...0704.00011Sparsity-certifying Graph Decompositions[0.014515653, 0.023809524, -0.028145121, -0.04...0704.00022The evolution of the Earth-Moon system based o...[-4.766115e-05, 0.017415706, 0.04146007, -0.03...0704.00033A determinant of Stirling cycle numbers counts...[0.027208889, 0.046175897, 0.0010913888, -0.01...0704.00044From dyadic $\Lambda_{\alpha}$ to $\Lambda_{\a...[0.0113909235, 0.0042667952, -0.0008565594, -0...0704.0005............99995Multiple Time Dimensions[0.02682626, -0.0015173098, -0.0019915192, -0....0812.386999996Depth Zero Representations of Nonlinear Covers...[-0.02740943, 0.011689809, -0.0105154915, -0.0...0812.387099997Decting Errors in Reversible Circuits With Inv...[0.0072460608, 0.0028085636, -0.015064359, -0....0812.387199998Unveiling the birth and evolution of the HII r...[0.009408689, -0.0047120117, 0.0021392817, -0....0812.387299999The K-Receiver Broadcast Channel with Confiden...[-0.0026305509, -0.006502139, 0.013400236, -0....0812.3873100000 rows × 3 columnsabstract_df=abstract_tables[1]abstract_df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>abstractembeddingsdoi0A fully differential calculation in perturba...[-0.035151865, 0.022851437, 0.025942933, -0.02...0704.00011We describe a new algorithm, the $(k,\ell)$-...[0.035485767, -0.0015772493, -0.0016615744, -0...0704.00022The evolution of Earth-Moon system is descri...[-0.014510429, 0.010210799, 0.049661566, -0.01...0704.00033We show that a determinant of Stirling cycle...[0.029191103, 0.047992915, -0.0061754594, -0.0...0704.00044In this paper we show how to compute the $\L...[-0.015174898, 0.01603887, 0.04062805, -0.0246...0704.0005............99995The possibility of physics in multiple time ...[0.016121766, 0.011126887, 0.018650021, -0.044...0812.386999996We generalize the methods of Moy-Prasad, in ...[-7.164341e-05, -0.007114291, -0.008979887, -0...0812.387099997Reversible logic is experience renewed inter...[0.03194286, -0.00771745, 0.015977046, -0.0474...0812.387199998Based on a multiwavelength study, the ISM ar...[-0.012340169, -0.021712925, 0.00806009, -0.00...0812.387299999The secrecy capacity region for the K-receiv...[0.0012416588, 0.0006933478, -0.0057888636, -0...0812.3873100000 rows × 3 columnsabstract_df['doi'].valuesarray(['0704.0001', '0704.0002', '0704.0003', ..., '0812.3871',
       '0812.3872', '0812.3873'], dtype=object)fromxvimportarxiv_urldoi=abstract_df['doi'].values[0]arxiv_url(doi)'https://arxiv.org/abs/0704.0001'fromxv.data_accessimportresource_descriptionsresource_descriptions{'abs': 'Main page of article. Contains links to all other relevant information.',
 'pdf': 'Direct link to article pdf',
 'format': 'Page giving access to other formats',
 'src': 'Access to the original source files submitted by the authors.',
 'cits': 'Tracks citations of the article across various platforms and databases.',
 'html': 'Link to the ar5iv html page for the article.'}doi='0704.0001'forresource,descriptioninresource_descriptions.items():print(f"{resource}:{description}")print(f"Example:{arxiv_url(doi,resource)}")print("")abs: Main page of article. Contains links to all other relevant information.
Example: https://arxiv.org/abs/0704.0001

pdf: Direct link to article pdf
Example: https://arxiv.org/pdf/0704.0001

format: Page giving access to other formats
Example: https://arxiv.org/format/0704.0001

src: Access to the original source files submitted by the authors.
Example: https://arxiv.org/src/0704.0001

cits: Tracks citations of the article across various platforms and databases.
Example: https://arxiv.org/cits/0704.0001

html: Link to the ar5iv html page for the article.
Example: https://ar5iv.labs.arxiv.org/html/0704.0001arxiv_url(doi,'pdf')'https://arxiv.org/pdf/0704.0001'pdf_bytes=g[arxiv_url(doi,'pdf')]The contents  (~1.647MB) of https://arxiv.org/pdf/0704.0001 are being downloaded...abstract_df.embeddings.values[0].shape(768,)