Package: notion-nlp
Notion NLPTo read text from a Notion database and perform natural language processing analysis.English/简体中文IntroductionTo achieve functionality similar to flomo, I have created a database using Notion, where I have recorded my thoughts and insights over the years, accumulating a rich corpus of language. However, the random roaming feature of flomo did not meet my needs, so I decided to develop a small tool that integrates with the Notion API and performs NLP analysis.Now, the tool can:Output intuitive and visually appealing word cloud images.Generate thematic summaries of your Notion notes.:memo:Example thematic summarySupport multiple languages. I have added stopword lists for several languages including Chinese, English, Russian, French, Japanese, and German. Users can also customize their own stopword lists.:earth_asia:Stopword lists for multiple languagesSupport multiple tasks. Users can configure multiple databases and corresponding filtering and sorting conditions to create rich analysis tasks.:mag:Example configuration fileFor example, I have added the following tasks::thinking: Reflections from the past year:triangular_flag_on_post: Optimization of annual summaries for the current year:warning: Self-admonitions from all time periodsI am pleased to share this tool and hope it can be helpful to you. :laughing:Piplineflowchart TB
A1[(Get personal access token from note-taking software)]
B2[Customize NLP module parameters]
B3[Customize visualization module parameters]
B1[Configure API key and corresponding database ID]
C1((Run task))
D1([Read rich text via API]) 
D2([Segment/Clean/Build word-sentence mapping])
D3[/Calculate TF-IDF/]
E1{{Markdown of keywords and source sentences}}
E2{{Word cloud with multiple color styles}}

  A1 -- Configure_task_parameters --> Parameter_types
  subgraph Parameter_types
  B1
  B2
  B3
  end
  B1 & B2 & B3 --> C1
  C1 --> Calculation_module
  subgraph Calculation_module
  D1 --> D2 --> D3
  end
  Calculation_module --> Visualization_module
  subgraph Visualization_module
  E1
  E2
  endInstallation and UsageWindows SystemDownloadthe latest releaseof theWindowsversionzipfile, extract it, and double-clickstart.batto follow the script prompts and start experiencing.Linux SystemMethod 1: Downloadthe latest releaseof theLinuxversionzipfile, extract it, open the terminal in the current directory, and enter./notion-nlp-linux --helpto view the command details.Method 2: Install the package fromPyPIto thePythonenvironment.pipinstallnotion-nlpConfigure TasksConfiguration file referenceconfig.sample.yaml(hereinafter config, please rename toconfig.yamlas your own configuration file)Get the integration tokenInnotion integrationscreate a new integration, get your own token and fill in the token in the config.yaml file afterwards.Graphic Tutorial:tango/markdownAdd integration to database/get database IDIf you open the notion database page in your browser or click on the share copy link, you will see the database id in the address link (similar to a string of jumbles) and fill in the database_id under the task of config.Graphic Tutorial:tango/markdownConfigure the filter sort database entry extra parameterThe task's extra is used to filter and sort the database, seenotion filter APIfor format and content, theconfig.sample.yamlfile already provides 2 configurations.Run all tasksSelect "Run all tasks" through Windows interactive scriptCall using terminal or Python code after installing the package from PyPIRun from command linepython3.8-mnotion_nlprun-all-tasks--config-file/path/to/your/config/fileRun from Python codefromnotion_nlpimportrun_all_tasksconfig_file="./notion-nlp-dataset/configs/config.yaml"run_all_tasks(config_file)Run a single taskSelect "Run specified task" through Windows interactive scriptCall using terminal or Python code after installing the package from PyPIIn therun_taskcommand, you can specify the task in several ways, including:task: an instance ofTaskParams;task_json: a JSON string representing the task information;task_name: the name of the task.Ifconfig_fileexists, you can usetask_nameto specify the task. Note that the task needs to be activated, otherwise an exception will be thrown. Ifconfig_filedoes not exist, you need to provide atokenand eitherTaskParamsortask_json.With an existingconfigfile, pass intask name/task json/task parameter classRun from command line# Option 1python3.8-mnotion_nlprun-task--task-nametask_1--config-file/path/to/your/config/file# Option 2python3.8-mnotion_nlprun-task--task-json'{"name": "task_1", "database_id": "your_database_id"}'--config-file/path/to/your/config/fileRun from Python codefromnotion_nlpimportrun_tasktask_name="task_1"database_id="your_database_id"config_file="./notion-nlp-dataset/configs/config.yaml"# Option 1run_task(task_name=task_name,config_file=config_file)# Option 2 (not recommended for Python code)importjsontask_info={"name":task_name,"database_id":database_id}run_task(task_json=json.dumps(task_info,ensure_ascii=False),config_file=config_file)# Option 3 (recommended)fromnotion_nlp.parameter.configimportTaskParamstask=TaskParams(name=task_name,database_id=database_id)run_task(task=task,config_file=config_file)Without aconfigfile, pass intokenandtask json/task parameter classRun from command line# Option 1python3.8-mnotion_nlprun-task--task-json'{"name": "task_1", "database_id": "your_database_id"}'--token'your_notion_integration_token'Run from Python codefromnotion_nlpimportrun_tasktask_name="task_1"database_id="your_database_id"notion_token="your_notion_integration_token"# Option 1 (not recommended for Python code)importjsontask_info={"name":task_name,"database_id":database_id}run_task(task_json=json.dumps(task_info,ensure_ascii=False),token=notion_token)# Option 2 (recommended)fromnotion_nlp.parameter.configimportTaskParamstask=TaskParams(name=task_name,database_id=database_id)run_task(task=task,token=notion_token)Enhance Personal Experience:customs: Custom Stopword ListAdd a text file in thestopwords directorywith the suffixstopwords.txt, such ascustom.stopwords.txt. Each stopword should be on a separate line in the file.:memo: Share Your Ideas with the AuthorJoin the discussionSubmit an issue:gift_heart: Buy the author a cup of coffee and request a personalized customization.DevelopmentWelcome to fork and add new features/fix bugs.After cloning the project, use thecreate_python_env_in_new_machine.shscript to create a Poetry virtual environment.After completing the code development, use the invoke command to perform a series of formatting tasks, including black/isort tasks added in task.py.invokecheckAfter submitting the formatted changes, run unit tests to check coverage.poetryruntoxNoteThe word segmentation tool has two built-in options: jieba/pkuseg. (Considering adding language analysis to automatically select the most suitable word segmentation tool for that language.)jieba is used by default.pkuseg cannot be installed with poetry and needs to be installed manually with pip. In addition, this library is slow and requires high memory usage. It has been tested that a VPS with less than 1G memory needs to load virtual memory to use it.The analysis method using tf-idf is too simple. Consider integrating the API of LLM (such as openAI GPT-3) for further analysis.Contributionsscikit-learn -https://github.com/scikit-learn/scikit-learnAlir3z4/stop-words -https://github.com/Alir3z4/stop-wordsLicense and CopyrightMIT LicenseThe MIT License is a permissive open-source software license. This means that anyone is free to use, copy, modify, and distribute your software, as long as they include the original copyright notice and license in their derivative works.However, the MIT License comes with no warranty or liability, meaning that you cannot be held liable for any damages or losses arising from the use or distribution of your software.By using this software, you agree to the terms and conditions of the MIT License.Contact informationSee more at myHomePage