Package: chunkgpt
ChunkGPT: Summarization for Large TextsChunkGPT is a lightweight Python library designed to assist users in summarizing long texts that exceed the token limit of OpenAI's GPT-based models. It breaks down large texts into manageable chunks, generates summaries for each chunk, and then combines these summaries into a coherent final summary.InstallationYou can install ChunkGPT using pip:pipinstallchunkgptDependenciesChunkGPT depends on the following Python libraries, which are both included in the install:LibraryVersionPurposeopenai0.27.8Accessing OpenAI and managing OpenAI API calls.tiktoken0.4.0Estimating token counts for a given string in order to appropriately apply chunk limits and determine OpenAI cost.UsageIt's easy to use ChunkGPT! Follow these general steps to generate summaries for your lengthy texts:fromchunkgpt.chunkgptimportChunker# Initialize the Chunker objectchunker=Chunker()# Load your text to be summarizedwithopen('myfile.txt','r')asinfile:mytext=infile.read()# Generate a summary for the textsummary=chunker.summarize(mytext)print(summary['result'])Advanced UsageThe behavior of ChunkGPT can be modified to suit the needs of your project. Below are some examples of advanced ways of modifying ChunkGPT.Customize the system messageThe defaultSYSTEM_MSGis the result of several rounds of trial and error during development of ChunkGPT, and has been shown to produce relatively reliable and detailed results.SYSTEM_MSG="""You are SUMMIFY, a specialized AI assistant\purpose-built to read long pieces of text and produce a concise\summary that greatly reduces the overall length of the text\without leaving out any critical details.Users will hand you an entire document, or just a portion of the\document, and it is your job to summarize the content in as few\words as possible, while conveying the essential meaning of the text.Do NOT refer to "the text" as an object. Instead, paraphrase the document you are\given so that your summary can essentially replace the document 1-for-1,\but with a significantly shorter length.The summary MUST be concise.The summary MUST be comprehensive.Include ALL KEY POINTS, and exclude any unnecessary excess."""However, if you are interested in experimenting with different system messages in order to tweak the quality or format of the final summary, you can define acustom_system_msgwhen initializing theChunkerobject.fromchunkgpt.chunkgptimportChunkersys_msg="Summarize the text you are given to the best of your ability."chunker=Chunker(custom_system_msg=sys_msg)Explicitly define your OpenAI API KeyBy default, ChunkGPT look for an environment variable calledOPENAI_API_KEY. If you choose to manage your API keys in a different way, you may define your API Key when initializing theChunkerobject.fromchunkgpt.chunkgptimportChunkermy_key='myApiKey1234'chunker=Chunker(api_key=my_key)Use a different modelChunkGPT supports allgpt-3.5andgpt-4models from OpenAI.fromchunkgpt.chunkgptimportChunkerchunker=Chunker(model='gpt-4')Tweak the temperatureBy default, ChunkGPT usees a temperature of 0. Temperatures closer to 0 will be less variable, while temperatures closer to 2 can be so variable as to be incomprehensible.fromchunkgpt.chunkgptimportChunkerchunker=Chunker(temperature=0.5)Tweak the token limits for chunks, their overlap, and the resulting summariesmax_chunk_lengthrepresents the max number of tokens used for each text chunk. Higher values may be more performant, but lower values may be more precise.chunk_overlaprepresents the number of tokens that overlap between consecutive segments. Lower values may be more performant, but higher values may smooth over transitions between chunks. This valuemust be smallerthanmax_chunk_length.summary_lengthrepresents the max number of tokens enforced for the output of the model. If your final summary result is getting cut off, you may want to consider increasing this value.fromchunkgpt.chunkgptimportChunkerchunker=Chunker(max_chunk_length=4000,summary_length=200,chunk_overlap=0)Note:summary_length+max_chunk_length+ the base number of tokens used by system and user promptsmust be smallerthan thetoken_limitof the model being used.Access Intermediate SummariesIf you wish to access the intermediate summaries, you can reference the'chunks'dictionary included in the response ofChunker.summarize().fromchunkgpt.chunkgptimportChunkerchunker=Chunker()text="..."summary=chunker.summarize(text)forchunkinsummary['chunks']:print(f"CHUNK{chunk}:")print("Original Text:")print(summary['chunks']['input'])print("Summary:")print(summary['chunks']['output'])Skip Final Reduction & Summarization StepBy default,Chunker.summarize()will reduce the final output until its length, combined with the system prompt andsummary_length, falls within the overalltoken_limit.However, if you find that this final summary is too short, and you want a more detailed summary, you can change this behavior by changing thefinal_stepparameter.This paramter is set to'summarize'by default, but it also accepts'combine', which instructs ChunkGPT to skip this final summary step and instead return a combination of all intermediate summaries.fromchunkgpt.chunkgptimportChunkerchunker=Chunker()text="..."summary=chunker.summarize(text,final_step='combine')Note: Because this final step is a simple concatenation of separately summarized excerpts, this option may lead to a more detailed, although possibly less coherent, final output.ChunkerObjectTheChunkerobject serves as the core mechanism responsible for breaking down large texts into smaller, manageable chunks, generating summaries for each of these chunks, and ultimately combining these chunk summaries into a coherent final summary.AttributesYou can customize theChunkerobject's behavior by including any of the following attributes as keyword arguments when initializing the object:AttributeTypeDefaultDescriptionmodelstr'gpt-3.5-turbo'Name of the OpenAI GPT modelapi_keystrOPENAI_API_KEYenvironment variableYourOpenAI API keymax_chunk_lengthint2048Maximum number of tokens for each text chunk (including any overlap from the previous chunk)chunk_overlapint50Number of tokens that overlap between consecutive chunkssummary_lengthint1024Maximum number of tokens for each summary (both intermediate summaries and final summary)temperaturefloat0.0Floating point number between 0 and 2, represents thetemperatureparameter used for each OpenAI completion.custom_system_msgstrNoneAlternative system message to use in place of the defaultSYSTEM_MSG.MethodsChunker.summarize(text)Generate and return a summary for a given text, along with additional data relevant to the summarizing process as a whole.ArgumentsArgumentTypeDescriptiontextstrThe text to be chunked and summarized.This method takes a string as input, breaks it into smaller chunks if necessary, generates summaries for each chunk, and then combines these summaries into one coherent final summary.If the combined length of the input text, ChunkGPT prompt, and the expected completion length does not exceed the model's token limit, a single summary will be produced.If the combined length of the intermediate summaries exceeds the model's token limit, the summaries will be chunked again, reducing the overall size of the summaries until the full summarized contents falls within the token limit.This method returns a dictionary with the following keys:KeyTypeDescription'original'strThe original, unaltered text represented as a string'result'strThe final resulting summary, comprehensive of all intermediate summaries.'chunks'dictA dictionary containing each chunk of the text, along with OpenAI's chat completion for that chunk. Each element in this dictionary is itself a dictionary containing the keys'input', representing the chunk text, and'output', representing the GPT-based Chat Completion for this chunk.'intermediate_steps'listA list containing a description of each intermediate step taken in the summarization process.'total_tokens'intThe total number of tokens consumed to generate this summary, including prompts and completions.'cost'floatThe estimated cost (in cents) for this completion, based on the model's price and the total tokens usedHelper MethodsThese methods are each responsible for a minor task within the summarization process. They are included here for reference, but you should rarely, if ever, need to access them directly.Chunker._chunk(text)Returns list of strings, each string representing a chunk of the original text that needs to be summarized. If the original text plus the system message and expected output is within the token limit, returns a single-item list containing the input text.Chunker._tokenize(text)Returns an encoded list of tokens from the given text string.Chunker._decode(tokens)Returns a decoded string from a list of tokens.Chunker._get_token_limit()Returns the token limit for each gpt-3.5 and gpt-4 model, current as of August 13, 2023.Chunker._get_price_per_token()Returns the price per token in cents for each gpt-3.5 and gpt-4 model, current as of August 13, 2023.Chunker._get_complete_token_count(text)Returns total number of tokens for a given input text, including the system prompt and expected number of tokens needed for the completion output.Chunker._get_base_token_count(text)Returns total number of tokens used by the system message and user message (prior to formatting with the input text).Chunker._construct_messages(text)Constructs a list of messages to submit to OpenAI for chat completion.Chunker._get_completion(text, num_retries=3)Uses the designated system_msg and the default USER_MSG template to get an OpenAI chat completion for the provided text.If the completion fails, ChunkGPT will retry up to 3 times, with a delay of 1 second between retries.ExamplesTheGithub repofor ChunkGPT inclues long sample .txt files to test out in theexamples/directory. Each file, when added to the existing system and user message templates, exceeds the token limit for all available OpenAI GPT models (as of August 13, 2023).You can test out the utility of this library against large texts by using these sample texts and employing any GPT models or Chunker object attributes you wish to experiment with.For example, see this script summarizing the 2023 IPCC Summary for Policymakers:fromchunkgpt.chunkgptimportChunkerchunker=Chunker()withopen('./examples/ipcc_summary_2023.txt')asinfile:text=infile.read()summary=chunker.summarize(text)output_str=f"""FINAL SUMMARY{summary['result']}TOTAL TOKENS IN ORIGINAL DOC{len(chunker._tokenize(summary['original']))}TOTAL TOKENS USED{summary['total_tokens']}MODEL USED{chunker.model}ESTIMATED COST${round(summary['cost']/100,2)}"""print(output_str)Running the script above, you should expect the output to look something like this:Initialized Chunker.
Split text into 16 chunks.
Got completion for chunk 1.
Got completion for chunk 2.
Got completion for chunk 3.
Got completion for chunk 4.
Got completion for chunk 5.
Got completion for chunk 6.
Got completion for chunk 7.
Got completion for chunk 8.
Got completion for chunk 9.
Got completion for chunk 10.
Got completion for chunk 11.
Got completion for chunk 12.
Got completion for chunk 13.
Got completion for chunk 14.
Got completion for chunk 15.
Got completion for chunk 16.
Split text into 2 chunks.
Got completion for chunk 1.
Got completion for chunk 2.
Got completion for combined summaries.
Reduced summary from 4462 to 2052.


FINAL SUMMARY
The IPCC's Climate Change 2023 Synthesis Report Summary for Policymakers emphasizes the urgent need for action to address greenhouse gas emissions and adapt to climate change. Key points include the role of human activities in global warming, the increase in global surface temperatures, the impact of well-mixed greenhouse gases, and the highest levels of CO2, methane, and nitrous oxide in millions of years. The report highlights the adverse impacts of climate change on ecosystems, water availability, food production, health, and infrastructure, as well as the barriers to adaptation. It discusses the projected increase in CO2 emissions, the gaps in current mitigation efforts, the costs and deployment of renewable energy technologies, and the uneven policy coverage for climate change. The report emphasizes the escalating risks and impacts of climate change, the need for adaptation, and the consequences for various regions and ecosystems. It underscores the importance of limiting global warming, the effectiveness of adaptation options, and the necessity of reaching net zero CO2 emissions. Delayed climate action will lead to increased global warming and more losses and damages, while accelerated climate action can provide co-benefits such as improved health and agricultural productivity. Ambitious mitigation pathways will require significant changes in economic structures and can be moderated by fiscal and regulatory reforms. Feasible, effective, and low-cost options for mitigation and adaptation are available, with differences across systems and regions. Net zero CO2 energy systems require a substantial reduction in fossil fuel use and widespread electrification. Rapid and far-reaching transitions across all sectors and systems are necessary for deep and sustained emissions reductions. Maintaining biodiversity and ecosystem services globally depends on effective conservation of approximately 30% to 50% of Earth's land, freshwater, and ocean areas. Cooperation with Indigenous Peoples and local communities is integral to successful adaptation and mitigation. Policy support for climate action is influenced by various actors in civil society, including businesses, youth, women, labor, media, Indigenous Peoples, and local communities. Finance, technology, and international cooperation are critical enablers for accelerated climate action, and closing the gaps in financial support for adaptation and mitigation is crucial, especially in developing countries. International cooperation is essential for ambitious climate action, including mobilizing finance, aligning finance flows with ambition levels, and supporting Nationally Determined Contributions (NDCs) and technology development and deployment.

KEY POINTS:
- Urgent action is needed to mitigate greenhouse gas emissions and adapt to climate change.
- Human activities are causing global warming, leading to an increase in global surface temperatures.
- Well-mixed greenhouse gases, such as CO2, methane, and nitrous oxide, are at their highest levels in millions of years.
- Climate change has adverse impacts on ecosystems, water availability, food production, health, and infrastructure.
- There are barriers to adaptation and gaps in current mitigation efforts.
- CO2 emissions are projected to increase, and renewable energy technologies need to be deployed.
- Climate change poses escalating risks and impacts, with consequences for various regions and ecosystems.
- Limiting global warming, effective adaptation options, and reaching net zero CO2 emissions are crucial.
- Delayed climate action will lead to increased global warming and more losses and damages.
- Accelerated climate action can provide co-benefits, such as improved health and agricultural productivity.
- Ambitious mitigation pathways require significant changes in economic structures and can be moderated by fiscal and regulatory reforms.
- Feasible, effective, and low-cost options for mitigation and adaptation are available, with differences across systems and regions.
- Net zero CO2 energy systems require a substantial reduction in fossil fuel use and widespread electrification.
- Rapid and far-reaching transitions across all sectors and systems are necessary for deep and sustained emissions reductions.
- Maintaining biodiversity and ecosystem services globally depends on effective conservation of land, freshwater, and ocean areas.
- Cooperation with Indigenous Peoples and local communities is integral to successful adaptation and mitigation.
- Policy support for climate action is influenced by various actors in civil society.
- Finance, technology, and international cooperation are critical enablers for accelerated climate action.
- Closing the gaps in financial support for adaptation and mitigation is crucial, especially in developing countries.
- International cooperation is essential for ambitious climate action, including mobilizing finance, aligning finance flows, and supporting Nationally Determined Contributions (NDCs) and technology development and deployment.

TOTAL TOKENS IN ORIGINAL DOCUMENT
31849

TOTAL TOKENS USED
44216

MODEL USED
gpt-3.5-turbo

ESTIMATED COST
$0.07ContributingContributions to improve ChunkGPT are welcome! If you find any issues or want to suggest enhancements, please submit an issue or pull request in theGitHub repository.LicenseThis project is licensed under theMIT License.AcknowledgementsChunkGPT was inspired by the need to summarize large texts efficiently using limited-token models. We thank the open-source community for their valuable contributions.Happy summarizing with ChunkGPT!