Package: azureml-rag
AzureML Retrieval Augmented Generation UtilitiesThis package is in alpha stage at the moment, use at risk of breaking changes and unstable behavior.It contains utilities for:Processing text documents into chunks appropriate for use in LLM prompts, with metadata such is source url.Embedding chunks with OpenAI or HuggingFace embeddings models, including the ability to update a set of embeddings over time.Create MLIndex artifacts from embeddings, a yaml file capturing metadata needed to deserialize different kinds of Vector Indexes for use in langchain. Supported Index types:FAISS index (via langchain)Azure Cognitive Search indexPinecone indexMilvus indexAzure Cosmos Mongo vCore indexGetting startedYou can install AzureMLs RAG package using pip.pipinstallazureml-ragThere are various extra installs you probably want to include based on intended use:faiss: When using FAISS based Vector Indexescognitive_search: When using Azure Cognitive Search Indexespinecone: When using Pinecone Indexesazure_cosmos_mongo_vcore: When using Azure Cosmos Mongo vCore Indexeshugging_face: When using Sentence Transformer embedding models from HuggingFace (local inference)document_parsing: When cracking and chunking documents locally to put in an IndexMLIndexMLIndex files describe an index of data + embeddings and the embeddings model used in yaml.Azure Cognitive Search Index:embeddings:dimension:768kind:hugging_facemodel:sentence-transformers/all-mpnet-base-v2schema_version:'2'index:api_version:2021-04-30-Previewconnection:id:/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace>/connections/<acs_connection_name>connection_type:workspace_connectionendpoint:https://<acs_name>.search.windows.netengine:azure-sdkfield_mapping:content:contentfilename:filepathmetadata:meta_json_stringtitle:titleurl:urlembedding:contentVectorindex:azureml-rag-test-206e03b6-3880-407b-9bc4-c0a1162d6c70kind:acsPinecone Index:embeddings:dimension:768kind:hugging_facemodel:sentence-transformers/all-mpnet-base-v2schema_version:'2'index:connection:id:/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace>/connections/<pinecone_connection_name>connection_type:workspace_connectionengine:pinecone-sdkfield_mapping:content:contentfilename:filepathmetadata:metadata_json_stringtitle:titleurl:urlindex:azureml-rag-test-206e03b6-3880-407b-9bc4-c0a1162d6c70kind:pineconeAzure Cosmos Mongo vCore Index:embeddings:dimension:768kind:hugging_facemodel:sentence-transformers/all-mpnet-base-v2schema_version:'2'index:connection:id:/subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace>/connections/<cosmos_connection_name>connection_type:workspace_connectionengine:pymongo-sdkfield_mapping:content:contentfilename:filepathmetadata:metadata_json_stringtitle:titleurl:urlembedding:contentVectordatabase:azureml-rag-test-dbcollection:azureml-rag-test-collectionindex:azureml-rag-test-206e03b6-3880-407b-9bc4-c0a1162d6c70kind:azure_cosmos_mongo_vcoreCreate MLIndexExamples using MLIndex remotely with AzureML and locally with langchain live here:https://github.com/Azure/azureml-examples/tree/main/sdk/python/generative-ai/ragConsume MLIndexfromazureml.rag.mlindeximportMLIndexretriever=MLIndex(uri_to_folder_with_mlindex).as_langchain_retriever()retriever.get_relevant_documents('What is an AzureML Compute Instance?')ChangelogPlease insert change log into "Next Release" ONLY.Next release0.2.25Using local cached encodings.Adding convert_to_dict() for openai v1.0+Check index_config before passing in validate_deployments.pyLimit size of documents upload to ACS in one batch to solve RequestEntityTooLargeError0.2.24.2Supporting*.cognitiveservices.*endpointAdding azureml-rag specific user_agent when using DocumentIntelligenceRefactored update index tasksSupporting uppercase file extensions name in crack_and_chunkFixing Deployment importing bug in utilsAdding the playgroundType tag in MLIndex Asset used for Azure AI studio playgroundRemove mandatory module-level imports of optional extra packages0.2.24.1Fixing is_florence key detectionUsing 'embedding_connection_id' instead of 'florence_connection_id' as parameter name0.2.24Introducing image ingestion with florence embedding APIAdding dummy output to validate_deployments for holding the right orderFixing DeploymentNotFound bug0.2.23.5Deprecate pkg_resources in logging.py (https://setuptools.pypa.io/en/latest/pkg_resources.html)0.2.23.4Make theapi_typeparameter non-case sensitive in OpenAIEmbedderBug fix in embeddings container path0.2.23.3Set upper bound forlangchainto 0.0.3480.2.23.2Make tiktoken pull from a cache instead of making the outgoing network call to get encodings filesAdd support for Azure Cosmos Mongo vCore0.2.23.1Fixing exception handling in validate_deployments to support OpenAI v1.0+0.2.23Support OpenAI v1.0 +Handle FAISS.load_local() change since Langchain 0.0.318Handle mailto links in url crawling component.Add support for Milvus vector store0.2.22update pypdf's version to 3.17.1 in document-parsing.0.2.21Use workspace connection tags instead of metadata since it's deprecated.Fix bug handling single files infiles_to_document_sources0.2.20Initial introduction of validate_deployments.Asset registration in *_and_register attempts to infer target workspace from asset_uri and handle multiple auth optionsactivity_logger moved out as first arg, this is an intermediate step as logger also shouldn't be first arg and instead handled by get_logger, activity_logger should be truly optional.validate_deployments itself was modified to make its interface closer to what existing tasks expect as input, and callable from other tasks as a function.0.2.19Introduce a newpathparameter in theindexsection of MLIndex documents over FAISS indices, to allow the path to FAISS index files to be different from the MLIndex document path.EnsureMLIndex.base_uriis never undefined for a valid MLIndex object.0.2.18.1Only save out metadata before embedding in crack_and_chunk_and_embed_and_indexUpdate create_embeddings to return num_embedded value.This enables crack_and_chunk_and_embed to skip loading EmbeddedDocument partitions when no documents were embedded (all reused).0.2.18Add new task to crack, chunk, embed, index to ACS, and register MLIndex in one step.Handleopenai.api_typebeingNone0.2.17Fix loading MLIndex failure. Don't need to get theendpointfrom connection when it is already provided.Try uselangchainVectorStore and fallback to vendorSupport `azure-search-documents==11.4.0b11``Add support for Pinecone in DataIndex0.2.16Use Retry-After when aoai embedding endpoint throws RateLimitError0.2.15.1Fix vendored FAISS langchain VectorStore to only error when a doc isNone(rather than when a Document isn't exactly the right class)0.2.15Support PDF cracking with Azure Document Intelligence servicecrack_and_chunk_and_embednow pulls documents through to embedding (streaming) and embeds documents in parallel batchesUpdate default field names.Fix long file name bug when writing to output during crack and chunk0.2.14Fix git_clone to handle WorkspaceConnections, again.0.2.13Fix git_clone to handle WorkspaceConnection objects and urls with usernames already in them.0.2.12Only process.jsonland.csvfiles when reading chunks for embedding.0.2.11Check casing for model kind and api_typeEnsure api_version not being set is supported and default make sense.Add support for Pinecone indexes0.2.10Fix QA generator and connections check for ApiType metadata0.2.9QA data generation accepts connection as input0.2.8Removeallowed_special="all"from tiktoken usage as it encodes special tokens like
as their special token rather then as plain text (which is the case when onlydisallowed_special=()is set on its own)Stop truncating texts to embed (to model ctx length) as newazureml.rag.embeddings.OpenAIEmbedderhandles batching and splitting long texts pre-embed then averaging the results into a single final embedding.Loosen tiktoken version range from~=0.3.0to<10.2.7Don't try and use MLClient for connections if azure-ai-ml<1.10.0Handle Custom Conenctions which azure-ai-ml can't deserialize today.Allow passing faiss index engine to MLIndex localPass chunks directly into write_chunks_to_jsonl0.2.6Fix jsonl output mode of crack_and_chunk writing csv internally.0.2.5Ensure EmbeddingsContainer.mount_and_load setscreate_destination=Truewhen mounting to create embeddings_cache location if it's not already created.Fixsafe_mlflow_start_runtoyield Nonewhen mlflow not availableHandle customfield_mappingspassed toupdate_acstask.0.2.4Introducecrack_and_chunk_and_embedtask which tracks deletions and reused source + documents to enable full sync with indexes, levering EmbeddingsContainer for storage of this information across Snapshots.Restoreworkspace_connection_to_credentialfunction.0.2.3Fix git clone url format bug0.2.2Fix all langchain splitter to use tiktoken in an airgap friendly way.0.2.1Introduce DataIndex interface for scheduling Vector Index Pipeline in AzureML and creating MLIndex AssetsVendor various langchain components to avoid breaking changes to MLIndex internal logic0.1.24.2Fix all langchain splitter to use tiktoken in an airgap friendly way.0.1.24.1Fix subsplitter init bug in MarkdownHeaderSplitterSupport getting langchain retriever for ACS based MLIndex with embeddings.kind: none.0.1.24Don't mlflow log unless there's an active mlflow run.Supportlangchain.vectorstores.azuresearchafterlangchain>=0.0.273upgraded toazure-search-documents==11.4.0b8Use tiktoken encodings from package for other splitter types0.1.23.2HandlePathobjects passed intoMLIndexinit.0.1.23.1Handle .api.cognitive style aoai endpoints correctly0.1.23Ensure tiktoken encodings are packaged in wheel0.1.22Set environment variables to pull encodings files from directory with cache key to avoid tiktoken external network callFix mlflow log error when there's no files input0.1.21Fix top level imports inupdate_acstask failing without helpful reason when oldazure-search-documentsis installed.0.1.20Fix Crack'n'Chunk race-condition where same named files would overwrite each other.0.1.19Various bug fixes:Handle some malformed git urls ingit_clonetaskTry fall back when parsing csv with pandas failsAllow chunking special tokensEnsure logging with mlflow can't fail a taskUpdate to support latestazure-search-documents==11.4.0b80.1.18Add FaissAndDocStore and FileBasedDocStore which closely mirror langchains' FAISS and InMemoryDocStore without the langchain or pickle dependency. These are default not used until PromptFlow support has been added.Pinazure-documents-search==11.4.0b6as there's breaking changes in11.4.0b7and11.4.0b80.1.17Update interactions with Azure Cognitive Search to use latest azure-documents-search SDK0.1.16Convert api_type from Workspace Connections to lower case to appease langchains case sensitive checking.0.1.15Add support for custom loadersAdded logging for MLIndex.initto understand usage of MLindex0.1.14Add Support for CustomKeys connectionsAdd OpenAI support for QA Gen and Embeddings0.1.13 (2023-07-12)Implement single node non-PRS embed task to enable clearer logs for users.0.1.12 (2023-06-29)Fix casing check of ApiVersion, ApiType in infer_deployment util0.1.11 (2023-06-28)Update casing check for workspace connection ApiVersion, ApiTypeint casting for temperature, max_tokens0.1.10 (2023-06-26)Update data asset registering to have adjustable output_typeRemove asset registering from generate_qa.py0.1.9 (2023-06-22)Addazureml.rag.data_generationmodule.Fixed bug that would cause crack_and_chunk to fail for documents that contain non-utf-8 characters. Currently these characters will be ignored.Improved heading extraction from Markdown files. Whenuse_rcts=FalseMarkdown files will be split on headings and each chunk with have the heading context up to the root as a prefix (e.g.# Heading 1\n## Heading 2\n# Heading 3\n{content})0.1.8 (2023-06-21)Add deployment inferring util for use in azureml-insider notebooks.0.1.7 (2023-06-08)Improved telemetry for tasks (used in RAG Pipeline Components)0.1.6 (2023-05-31)Fail crack_and_chunk task when no files were processed (usually because of a malformedinput_glob)Changeupdate_acs.pyto defaultpush_embeddings=Trueinstead ofFalse.0.1.5 (2023-05-19)Add api_base back to MLIndex embeddings config for back-compat (until all clients start getting it from Workspace Connection).Add telemetry for tasks used in pipeline components, not enabled by default for SDK usage.0.1.4 (2023-05-17)Fix bug where enabling rcts option on split_documents used nltk splitter instead.0.1.3 (2023-05-12)Support Workspace Connection based auth for Git, Azure OpenAI and Azure Cognitive Search usage.0.1.2 (2023-05-05)Refactored document chunking to allow insertion of custom processing logic0.0.1 (2023-04-25)Features AddedIntroduced packagelangchain Retriever for Azure Cognitive Search