Package: jarvis-nlp
Natural Language User InterfacePlatform SupportedLanguage StatsRepo StatsCode StatsDeploymentsActivityDevelopment and MaintenanceReach OutKick offInstallpython-mpipinstalljarvis-ironmanInitiateimportjarvisif__name__=='__main__':jarvis.start()PrerequisitesMacOSTested onmacOS High Sierra, Mojave, Catalina, Big Sur, Monterey and Ventura*System Preferences→Security & Privacy→PrivacyClick+sign and add the preferredIDEandTerminalin the following sections in left pane.Microphone-Requiredto listen and respond.Accessibility-Requiredto use key combinations for brightness and volume controls.Camera-[Optional]Required only during face recognition/detection.Automation-Requiredto controlSystem Eventsand other apps like Outlook and Calendar.Files and Folders[OR]Full Disk Access-Requiredfor allFileIOoperations.:warning: Known Issue withpyttsx3 moduleonmacOS Ventura 13.0: This version of macOS does not hold the attributeVoiceAge.Workaround has been raised as a PRLinuxTested onUbuntu 22.04 LTSStore the host machine's password as the env var:ROOT_PASSWORDUnlike macOS and Windows,Ubuntudoes not have app specific permissions.WindowsTested onWindows 10Settings→PrivacyMicrophone-Requiredto listen and respond.Camera-[Optional]Required only during face recognition/detection.UnlikemacOS,Windowspops a confirmation window toAlloworDenyaccess to files and folders.InstallAnacondaorMiniconda,VisualStudio C++ BuildTools, andGitMake sure C++ build tools are installed completely and restartAdd anaconda/miniconda scripts location toPATHin Environment VariablesTest peripheralsCamerafromjarvis.modules.cameraimportcameraif__name__=='__main__':cam_object=camera.Camera()print(cam_object.get_index())# Get index id of each cameraprint(cam_object.list_cameras())# List all connected camerasText to Speechfromjarvis.modules.speakerimportspeakif__name__=='__main__':speak_object=speak.Speaker()speak_object.run()print(list(speak_object.get_all_voices()))Speech to Textimportasynciofromjarvis.modules.microphoneimportrecognizerif__name__=='__main__':asyncio.run(recognizer.main())ENV VariablesEnvironment variables are loaded from a.envfile and validated usingpydanticMore on Environment variablesROOT_PASSWORD- System password to get the system vitals and run othersudocommands. Mandatory forLinuxNAME- Name which Jarvis should address the user by. Defaults toVigneshTITLE- Title which Jarvis should address the user by. Defaults tosirPLOT_MIC- Boolean value whether to show microphone usage in realtime. Defaults toTrueLOG_RETENTION- Number of days to store the logs. Defaults to10WAKE_WORDS- List of wake words to initiate Jarvis' listener. Defaults to['jarvis'](Defaults to['alexa']in legacy macOS):warning: Jarvis has limitations on the wake words as it relies on ML libraries for wake word detection.VOICE_NAME- Name of the voice supported by the OperatingSystem. Defaults to the author's favorite.VOICE_RATE- Speed/rate at which the text should be spoken. Defaults to the value frompyttsx3module. Typically200To add more voicesmacOS:System Preferences → Accessibility → Spoken Content → System voice → Manage Voices...Windows:Settings → Time & Language → Speech → Manage voices → Add voicesSENSITIVITY- Hot word detection sensitivity. Allowed range: [0-1] Defaults to0.5TIMEOUT- Timeout in seconds until which the listener should wait for speech. Defaults to3PHRASE_LIMIT- Timeout in seconds until which the listener will remain active. Defaults toNoneLIMITED- Boolean flag to run only the main version ofJarvisskipping background processes. Defaults toFalseEnforced based on the number of CPU cores.DEBUG- Boolean flag to enable debug level for logging. Defaults toFalseRECOGNIZER_SETTINGS- A JSON object that has with customized speech recognition settings.Custom settings for speech recognitionThese are customized according to the author's voice pitch.
Please userecognizer.pyto figure out the suitable values on a trial and error basis.These settings are added (optionally), to avoid the hard codedPHRASE_LIMITCons in using hard codedPHRASE_LIMIT:Disables the listener after the set limit even the speaker is actively talking.Listener will be active until the set limit even after the speaker has stopped talking.Sample settings (formatted as JSON object)RECOGNIZER_SETTINGS:'{"energy_threshold": 1100, "dynamic_energy_threshold": false, "pause_threshold": 2, "phrase_threshold": 0.1, "non_speaking_duration": 2}'Descriptionenergy_threshold: Minimum audio energy to consider for recording. Greater the value, louder the speech should be.dynamic_energy_threshold: Change considerable audio energy threshold dynamically.pause_threshold: Seconds of non-speaking audio before a phrase is considered complete.phrase_threshold: Minimum seconds of speaking audio before it can be considered a phrase - values below this are ignored. This helps to filter out clicks and pops.non_speaking_duration: Seconds of non-speaking audio to keep on both sides of the recording.PeripheralsCAMERA_INDEX- Camera index that has to be used. Runcamera.pyto get the index value of each camera.SPEAKER_INDEX- Speaker index that has to be used. Runperipherals.pyto get the index value of each speaker.MICROPHONE_INDEX- Microphone index that has to be used. Runperipherals.pyto get the index value of each microphone.FeaturesGIT_USER- GitHub UsernameGIT_PASS- GitHub TokenWEATHER_API- API Key fromopenweathermapNEWS_API- API Key fromnewsapiMAPS_API- API Key for maps fromGoogleBIRTHDAY- Birth date in the format DD-MM - Example:24-AprilWOLFRAM_API_KEY- API Key from wolfram alpha.Email/SMS notificationsGMAIL_USER- Gmail account username to send and read emails.GMAIL_PASS- Gmail account password to send and read emails.ALT_GMAIL_USER- Alternate gmail account username to send an SMS. (GMAIL_USERcan be re-used)ALT_GMAIL_PASS- Alternate gmail account password to send an SMS. (GMAIL_PASScan be re-used)RECIPIENT- Email address to which the emails from jarvis have to be received.iOS integrationsICLOUD_USER- iCloud account username/email.ICLOUD_PASS- iCloud account password.ICLOUD_RECOVERY- Recovery phone number to activate lost mode on a target device - Example:+11234567890PHONE_NUMBER- To send SMS from Jarvis - Example:+11234567890Calendar/Meeting integrationsICS_URL- Shared calendar URL to get meetings information from. Should end with.icsEVENT_APP- To read events fromoutlookorcalendarapplication inmacOS. Defaults tocalendar:bulb:   Whencalenderis used, the name of thecalendarwithin theCalendar.appshould beJarvisBackground scans [Defaults to 1 hour]SYNC_MEETINGS- Interval in seconds to generatemeetingsinformation using anicsURL.SYNC_EVENTS- Interval in seconds to generateeventsinformation usingcalendaroroutlookapplication.Wi-Fi ControlsWIFI_SSID- SSID of the wireless connection.WIFI_PASSWORD- Password for the wireless connection.CONNECTION_RETRY- Frequency in seconds to check for an active internet connection. Defaults to 10 seconds.VPNServerintegrationVPN_USERNAME- Username to create vpn-server. Defaults to profile username.VPN_PASSWORD- Password to authenticate vpn-server. Defaults to profile password.VPN_DOMAIN- Domain name for the hosted zone.VPN_RECORD_NAME- Alias record name to access VPN server.Car Controls- Applies only for JLR vehicles subscribed toInControlapplication.CAR_EMAIL- Email address to log in to InControl API.CAR_PASS- Password to authenticate InControl API.CAR_PIN- InControl PIN.Garage Controls- Applies only for garages usingMyQ garage controller.MYQ_USERNAME- Email address to log in to MyQ API.MYQ_PASSWORD- Password to authenticate MyQ API.Telegram BotintegrationBOT_TOKEN- Telegram BOT token.BOT_CHAT_IDS- UserID/ChatID for a particular user.BOT_USERS- Usernames that should have access to Jarvis.OS Agnostic Voice ModelSPEECH_SYNTHESIS_TIMEOUT- Timeout to connect to the docker container that processes text to speech requests.SPEECH_SYNTHESIS_VOICE- Voice for the speech synthesis model. Defaults to author's favorite.SPEECH_SYNTHESIS_QUALITY- Quality of speech synthesis conversion. Defaults tomedium.To enable independent speech-synthesisdockerrun\-it\-p5002:5002\-e"HOME=${HOME}"\-v"$HOME:${HOME}"\-v/usr/share/ca-certificates:/usr/share/ca-certificates\-v/etc/ssl/certs:/etc/ssl/certs\-w"${PWD}"\--user"$(id-u):$(id-g)"\rhasspy/larynx:bulb:   Speech Synthesis can run on a docker container for better voices but, response might be negligibly slower. If you don't have docker installed or simply don't want to use it, set theSPEECH_SYNTHESIS_TIMEOUTenv var to 0. This is also done automatically if failed to launch a docker container upon startup.Offline communicatorOFFLINE_PORT- Port number to initiate offline communicator. Defaults to4483OFFLINE_PASS- Secure phrase to authenticate offline requests. Defaults toOfflineCommWORKERS- Number of uvicorn workers (processes) to spin up. Defaults to1Stock PortfolioROBINHOOD_USER- Robinhood account username.ROBINHOOD_PASS- Robinhood account password.ROBINHOOD_QR- Robinhood loginQR codeAPI FeaturesROBINHOOD_ENDPOINT_AUTH- Authentication token to access the robinhood portfolio which is generated every hour.SURVEILLANCE_ENDPOINT_AUTH- Token to access webcam live feed via Jarvis API.SURVEILLANCE_SESSION_TIMEOUT- Session time out for/surveillance. Defaults to 300 seconds.Background TasksThere are two options to run background tasks on Jarvis.Jarvis can run internal tasks (offline communicator compatible) at certain intervals using abackground_tasks.yamlfile stored infileiodirectory.Setup InstructionsThis is the sample content ofbackground_tasks.yaml-seconds:1_800task:just turn off all lights# Runs every 30 minutes - 'just' flag retains the lights' last setting (eg: brightness or color)-seconds:10_800task:remind me to drink water# Runs every 3 hours ignoring the hours specifiedignore_hours:-21-22-23-0-1-2-3-4-5-6CRONTAB- Runs external tasks using cron expressions. Needs to be stored as env var.Sample value["00**1-5/2find/var/log-delete","05**1tar-zcf/var/backups/home.tgz/home/"]ContactsJarvis can send on demand notifications using acontacts.yamlfile stored infileiodirectory. Usesgmail-connectorfor SMS and email notifications.Setup InstructionsNote: Jarvis currently supports sending emails only when thecontacts.yamlfile is present, however phone numbers can be used directly.phone:Tony:0123456789Thor:1234567890email:Eddard:ned@gmail.comAegon:egg@yahoo.comSmart DevicesA source filesmart_devices.yamlis used to store smart devices' hostnames.
Jarvis supportsMagicHomeforlights,LGWebOSandRokufor TVs.TV hostnames should include the brand name [LG/Roku] to distinguish the modules accordingly.This will be set by default, if yours doesn't include the brand name change it in the TV settings.To wake upRokuTVs using MAC address, make sure theBandwidth saverfeature is turned off under,Settings/Network/Bandwidth saver >> OffFor first time users onLGWebOSTVs, there will be a prompt on the TV to accept the connection request.Once the connection request is accepted a client key will be generated and logged.Please make sure to store this insmart_devices.yamlfile to avoid repeated connection prompt.Setup InstructionsTV identifiers should have the wordtvto distinguish between lights and tv.The name used in the keys (for both lights and tv) will be the identifier when an action is requested.Lights should be a dictionary of identifier and a list of hostnames.TVs should be a nested dictionary of multiple parameters.The source file (smart_devices.yaml) should be as following:bedroom:-'HOSTNAMES'hallway:-'HOSTNAMES'hallway basement:-'HOSTNAMES'kitchen:-'HOSTNAMES'living room:-'HOSTNAMES'party mode:# Light hostnames that needs to be engaged for party mode, if not present individual lights can be enabled-'HOSTNAMES'living room tv:hostname:'HOSTNAME'client_key:'CLIENT_KEY'# Generated automatically if unavailablemac_address:# Mac addresses can either be a string or a list-'WIRED_MAC_ADDRESS'-'WIRELESS_MAC_ADDRESS'bedroom tv:hostname:'HOSTNAME'mac_address:'MAC_ADDRESS'Automation Setup [Optional]Jarvis can executeoffline compatibletasks
at pre-defined times without any user interaction. Uses anautomation.yamlfile as source which should be stored
within the directoryfileioSetup InstructionsThe YAML file should be a dictionary within a dictionary that looks like the below.OPTIONAL:The key,daycan be alistof days, or astrof a specific day or simply astrsayingweekdayorweekendwhen the particular automation should be executed.Not having the keydaywill run the automation daily.
Date format should match exactly as described below.06:00 AM:day:weekday# Runs only between Monday and Fridaytask:set my bedroom lights to 50%06:30 AM:day:# Runs only on Monday, Wednesday and Friday-Monday-wednesday-FRIDAYtask:set my bedroom lights to 100%08:00 AM:# Runs only on Saturday and Sundayday:weekendtask:set my bedroom lights to 100%09:00 PM:# Runs dailytask:set my bedroom lights to 5%12:00 AM:# Even performs tasks that are not supported via voice commandstask:restart all background processesSimulation Setup [Optional]Jarvis can executeoffline compatibletasks
as a simulation to test the required functions and send an email with the results. Uses asimulation.yamlfile as source which should be stored
within the directoryfileioSetup InstructionsThe YAML file should be a list of phrases within a dictionary that looks like the below.meeting_event:-get me the events from my calendar-what meetings do I have todayGuidePlease refer to thewikipage for API usage, access controls, env variables, features' overview and demo videos.FAQsPlease refer to theFAQssection of the wiki.Coding StandardsDocstring format:GoogleStyling conventions:PEP 8Clean code with pre-commit hooks:flake8andisortRelease NotesRequirementpython-mpipinstallchangelog-generatorUsagechangelogreverse-frelease_notes.rst-t'Release Notes'LintingPreCommitwill ensure linting, and the doc creation are run on every commit.Requirementpipinstallsphinx==5.1.1pre-commitrecommonmarkUsagepre-commitrun--all-filesPypi Packagehttps://pypi.org/project/jarvis-ironman/Runbookhttps://thevickypedia.github.io/Jarvis/License & copyright© Vignesh Sivanandha RaoLicensed under theMIT License