Package: emailproxy
Email OAuth 2.0 ProxyTransparently add OAuth 2.0 support to IMAP/POP/SMTP client applications, scripts or any other email use-cases that don't support this authentication method.Motivation and capabilitiesEmail services that support IMAP, POP and/or SMTP access are increasingly requiring the use of OAuth 2.0 to authenticate connections, but not all clients support this method.
This tool creates a simple local proxy that intercepts the traditional IMAP/POP/SMTP authentication commands and transparently replaces them with the appropriate SASL (X)OAuth 2.0 commands and credentials.
Your email client can continue to use theloginorauth/authenticateoptions, with no need to make it aware of OAuth's existence.
The proxy works in the background with a menu bar/taskbar helper or as a headless system service, and is compatible with macOS, Windows and Linux.Example use-casesYou need to use an Office 365 email account, but don't get on with Outlook.
The email client you like doesn't support OAuth 2.0, which became mandatoryin January 2023.You used to use Gmail via IMAP/POP/SMTP with your raw account credentials (i.e., your real password), but cannot do this now that Google has disabled this method, and don't want to use anApp Password(or cannot enable this option).You have an account already set up in an email client, and you need to switch it to OAuth 2.0 authentication.
You can edit the server details, but the client forces you to delete and re-add the account to enable OAuth 2.0, and you don't want to do this.You have made your own script or application that sends or receives email, but it doesn't support OAuth 2.0, and you don't want to have to modify it to implement this.You work with multiple services or applications that use IMAP/POP/SMTP, and you don't want to have to set up OAuth 2.0 independently for each one.In all of these cases and more, this proxy can help.
Follow the instructions here to get started, and pleaseopen an issuewith any problems or suggestions.
For commercial support or feature requests, please also considersponsoring this project.Getting startedBegin by downloading the proxy via one of the following methods:Pick apre-built releasefor your platform (macOS or Windows; no installation needed);or,Install fromPyPI: set up usingpython -m pip install emailproxy[gui], download thesampleemailproxy.configfile, thenpython -m emailproxyto run;or,Clone ordownload(and star :-) theGitHub repository, then:python -m pip install -r requirements-core.txt -r requirements-gui.txtto install requirements, andpython emailproxy.pyto run.Next, edit the sampleemailproxy.configfile to add configuration details for each email server and account that you want to use with the proxy.Guidance and example account configurationsare provided for Office 365, Gmail and several other providers, though you will need to insert your own client credentials for each one (see theclient credentials documentationfor guidance).
You can remove details from the sample configuration file for services you don't use, or add additional ones for any other OAuth 2.0-authenticated IMAP/POP/SMTP servers you would like to use with the proxy.You can now start the proxy: depending on which installation option you chose, either launch the application or use the appropriate run command listed above.
A menu bar/taskbar icon should appear.
If this does not happen, see thedependencies and setupsection for help resolving this.
For additional options, including fully headless deployments and integration with a secrets manager, see theoptional argumentsandadvanced configurationsections.Finally, open your email client and configure its server details to match the ones you set in the proxy's configuration file.
The correct server to use with an account is identified using the port number you select in your client – for example, to use the sample Office 365 details, this would be127.0.0.1on port1993for IMAP, port1995for POP and port1587for SMTP.
The proxy supports multiple accounts simultaneously, and all accounts associated with the same provider can share the same proxy server.
The local connection in your email client should be configured as unencrypted to allow the proxy to operate, but the connection between the proxy and your email server is always secure (implicit SSL/TLS for IMAP and POP; implicit or explicit (STARTTLS) SSL/TLS for SMTP).
See thesample configuration filefor additional documentation about advanced features, including local encryption, account configuration inheritance and support for running in a container.The first time your email client makes a request you should see a notification from the proxy about authorising your account.
Click the proxy's menu bar icon, select your account name in theAuthorise accountsubmenu, and then log in via the popup browser window that appears.
The window will close itself once the process is complete.
See the variousoptional argumentsfor support completing authentication if running without a GUI.After successful authentication and authorisation you should have IMAP/POP/SMTP access to your account as normal.
Make sure you keep the proxy running at all times to allow it to authorise your email client's background activity – enableStart at loginfrom the proxy's menu, or see theauto-start instructionsabout how to configure this in various different setups.After your accounts are fully set-up and authorised, no further proxy interaction should be required unless your account needs authorising again.
It will notify you if this is the case.OAuth 2.0 client credentialsAs part of the proxy setup process you need to provide an OAuth 2.0client_idandclient_secretto allow it to authenticate with email servers on your behalf.If you have an existing client ID and secret for a desktop app, you can use these directly in the proxy.
If this is not possible, you can also reuse the client ID and secret from any email client that supports IMAP/POP/SMTP OAuth 2.0 authentication with the email server you would like to connect to (such asthevariousopensourceclientswith OAuth 2.0 support), but please do this with care and restraint as access through reused tokens will be associated with the token owner rather than your own client.If you do not have access to credentials for an existing client you will need to register your own.
The process to do this is different for each provider, but the registration guides for several common ones are linked here.
In all cases, when registering, make sure your client is set up to use an OAuth scope that will give it permission to access IMAP/POP/SMTP as desired.
It is also highly recommended to use a scope that will grant "offline" access (i.e., a way torefresh the OAuth 2.0 authentication tokenwithout user intervention).
Thesample configuration fileprovides example scope values for several common providers.Office 365: register a newMicrosoft identity applicationGmail / Google Workspace: register aGoogle API desktop app clientAOL and Yahoo Mail (and subproviders such as AT&T) are not currently allowing new client registrations with the OAuth email scope – the only option here is to reuse the credentials from an existing client that does have this permission.The proxy supportsGoogle Cloud service accountsfor access to Google Workspace Gmail.
It also supports theclient credentials grant (CCG)andresource owner password credentials grant (ROPCG)OAuth 2.0 flows.
Please note that currently only Office 365 is known to support the CCG and ROPCG methods.
See thesample configuration filefor further details.Optional arguments and configurationWhen starting the proxy there are several optional arguments that can be set to customise its behaviour.--no-guiwill launch the proxy without an icon, which allows it to be run as asystemctlservice as demonstrated inthis example, or fully headless as demonstrated invariousothersubprojects.
Please note that unless you also specify one of the authorisation options below, this mode is only of use if you have already authorised your accounts through the proxy in GUI mode, or are importing a pre-authorised proxy configuration file from elsewhere.
If you do not set--external-author--local-server-auth, accounts that have not yet been authorised (or for whatever reason require re-authorisation) will time out when authenticating, and an error will be printed to the log.--external-authconfigures the proxy to present an account authorisation URL to be opened in an external browser and wait for you to copy+paste the post-authorisation result.
In GUI mode this can be useful in situations where the proxy's own browser window does not have access to some required authentication attribute of your typical setup.
In no-GUI mode this option allows you to authenticate accounts entirely externally (unlike--local-server-auth, which starts a local web server), though you will need to monitor the proxy's output and/or log for authentication notifications.After visiting the link provided and authorising account access, paste the final URL from your browser's address bar back into the proxy's popup window (GUI mode) or the terminal (no-GUI mode) to give it access to transparently proxy your login.
You should ignore any browser error message that is shown (e.g.,unable to connect); the important part is the URL itself.
This argument is identical to enabling external authorisation mode from theAuthorise accountsubmenu of the proxy's menu bar icon.--local-server-authis similar to--external-auth, but instead instructs the proxy to temporarily start an internal web server to receive authentication responses.
The--external-authoption is ignored in this mode.
To authorise your account, visit the link that is provided, authenticate, and proceed until you are presented with a success webpage from the proxy.
Please note that while authentication links can actually be visited from anywhere to log in and authorise access, by default the final redirection target (i.e., a link starting with your account'sredirect_urivalue) must be accessed from the machine hosting the proxy itself so that the local server can receive the authorisation result.
See thesample configuration filefor advanced options to configure this (viaredirect_listen_address).--config-fileallows you to specify the location of aconfiguration filethat the proxy should load.
If this argument is not provided, the proxy will look foremailproxy.configin its working directory.
By default, the proxy also saves cached OAuth 2.0 tokens back to this file, so it must be writable.
See the--cache-storeoption, if you would rather store configuration and cached values separately.--cache-storeis used to specify a separate location in which to cache authorised OAuth 2.0 tokens and associated metadata.
The value of this argument can either be the full path to a local file (which must be writable), or an identifier for an external store such as a secrets manager (see theadvanced configurationsection).
If this argument is not provided, credentials will be cached in the current configuration file.--log-fileallows you to specify the location of a file to send log output to (full path required).
Log files are rotated at 32MB and 10 older log files are kept.
This option overrides the proxy's default behaviour, which varies by platform (seebelowfor details).--debugenables debug mode, printing more verbose output to the log asdiscussed below.
This argument is identical to enabling debug mode from the proxy's menu bar icon.Advanced configurationTheexample configuration filecontains further documentation for various additional features of the proxy, including catch-all (wildcard) accounts, locally-encrypted connections and advanced Office 365 OAuth 2.0 flows.The proxy caches authenticated OAuth 2.0 tokens and associated metadata back to its own configuration file by default, but can alternatively be configured to use either a separate local file or a secrets manager service for this purpose.
Currently onlyAWS Secrets Manageris supported for remote token storage.
To use this feature, set the--cache-storeparameter to either a full ARN or a secret name, prefixing the value withaws:to identify its type to the proxy.
You must also install the AWS SDK for Python:python -m pip install boto3andset up authentication credentials(including a region).
The minimum required permissions for the associated AWS IAM user aresecretsmanager:GetSecretValueandsecretsmanager:PutSecretValue.
If the named AWS Secret does not yet exist, the proxy will attempt to create it; here, thesecretsmanager:CreateSecretpermission is also required.If you are using the proxy in a non-GUI environment it is possible to skip installation of dependencies that apply only to the interactive version.
To do this, install viapython -m pip install emailproxy(i.e., without the[gui]variant option), and pass the--no-guiargument when starting the proxy.
Please note that the proxy was designed as a GUI-based tool from the outset due to the inherently interactive nature of OAuth 2.0 authorisation, and there are limits to its ability to support fully no-GUI operation.
See theoptional arguments and configurationsection of this file for further details.If your network requires connections to use an existing proxy, you can instruct the script to use this by setting theproxy handlerenvironment variablehttps_proxy(and/orhttp_proxy) – for example,https_proxy=localhost python -m emailproxy.After installing its requirements, the proxy script can be packaged as a single self-contained executable usingpyinstallerif desired:pyinstaller --onefile emailproxy.py.
If you are using the GUI version of the proxy, you may need to add--hidden-import timeago.locales.en_shortuntilthistimeagoissueis resolved.Python 3.6 or later is required to run the proxy.
Thepython2 branchprovides minimal compatibility with python 2.7, but with a limited feature set, and no ongoing maintenance.
Seeissue 38for further discussion.Starting the proxy automaticallyIn order for the proxy to authenticate background requests from your email client it needs to be kept running constantly.
The easiest way to do this is to start the script automatically.
In GUI mode the proxy has basic support for this built-in: click its menu bar icon and then selectStart at login, which will stop the terminal instance and restart the script, configuring it to run each time you log in.
On macOS, if you are presented with a prompt about file access here, make sure you grant this so that python can run the proxy in the background.
For more advanced configurations, you may want to customise the startup behaviour and edit the script's parameters.
The method to achieve this differs depending on whether you are using macOS, Windows or Linux.On macOS, the file~/Library/LaunchAgents/ac.robinson.email-oauth2-proxy.plistis used to configure automatic starting of the proxy.
If you stop the proxy's service (i.e.,Quit Email OAuth 2.0 Proxyfrom the menu bar), you can restart it usinglaunchctl start ac.robinson.email-oauth2-proxyfrom a terminal.
You can stop, disable or remove the service from your startup items either via the menu bar icon option, or usinglaunchctl unload [plist path].
If you edit the plist file manually, make sure youunloadand thenloadit to update the system with your changes.
If theStart at loginoption appears not to be working for you on macOS, see theknown issues sectionfor potential solutions.On Windows the auto-start functionality is achieved via a shortcut in your user account's startup folder.
Pressing⊞ Win+rand enteringshell:startup(and then clicking OK) will open this folder – from here you can either double-click theac.robinson.email-oauth2-proxy.cmdfile to relaunch the proxy, edit it to configure, or delete this file (either manually or by deselecting the option in the proxy's menu) to remove the proxy from your startup items.On Linux this feature assumes that your system supports XDG Autostart.
A Desktop Entry fileac.robinson.email-oauth2-proxy.desktopwill be created in~/.config/autostart/.
Use the proxy's menu option (or manually remove this file) to prevent it starting when you log in.
It is also possible to run the proxy as a service (e.g., viasystemctl) – see the--no-guimode option above for more details.TroubleshootingIf you encounter problems using the proxy, enablingDebug modefrom the menu or passing--debugas a command line argument will print all client–proxy–server communication to your system log to help identify the issue.
This will include all commands to and responses from the server (and also as a result the content of your email messages).On macOS this can be viewed using Console.app orlog stream --predicate 'subsystem == "ac.robinson.email-oauth2-proxy"' --level=debug.
On Windows a fileemailproxy.logwill be created in the same location as the proxy (see also the--log-fileoption).
On Linux you can use, for example,tail -f /var/log/syslog | grep "Email OAuth 2.0 Proxy".Please note that debug mode may also result in your login credentials being printed to the log (though this is avoided where possible).
However, it is worth pointing out that because account authorisation with the remote email server is handled entirely through OAuth 2.0 in a web browser, while the username you set in your email client must be correct, the password used for the local IMAP/POP/SMTP connection to the proxy can be anything you like, and does not need to be the one you actually use to log in to your account (though it must be the same value each time, or you will be asked to re-authenticate repeatedly by the proxy).
The password you provide via your email client is used only to encrypt and decrypt the OAuth 2.0 authentication token that the proxy transparently sends to the server on your behalf.
Because of this, if you are concerned about debug mode and security you can use a test password for debugging and then replace it with a secure password (and authenticate again) once set up.It is often helpful to be able to view the raw connection details when debugging (i.e., without using your email client).
This can be achieved usingtelnet,PuTTYor similar.
For example, to test the Office 365 IMAP server from theexample configuration, first open a connection usingtelnet 127.0.0.1 1993, and then send a login command:a1 login e@mail.com password, replacinge@mail.comwith your email address, andpasswordwith any value you like during testing (see above for why the password is irrelevant).
If you have already authorised your account with the proxy you should see a response starting witha1 OK; if not, this command should trigger a notification from the proxy about authorising your account.If you are using asecure local connectionthe interaction with the remote email server is the same as above, but you will need to use a local debugging tool that supports encryption.
The easiest approach here is to useOpenSSL:openssl s_client -crlf -connect 127.0.0.1:1993.If you are having trouble actually connecting to the proxy, it is always worth double-checking thelocal_addressvalues that you are using.
Thesample configuration filesets this parameter to127.0.0.1for all servers.
If you remove this value and do not provide your own, the proxy defaults to::– in most cases this resolves tolocalhostfor both IPv4 and IPv6 configurations, but it is possible that this differs depending on your environment.
If you are unable to connect to the proxy from your client, it is always worth first specifying this value explicitly – see thesample configuration filefor further details about how to do this.
Please try setting and connecting to both IPv4 (i.e.,127.0.0.1) and IPv6 (i.e.,::1) loopback addresses before reporting any connection issues with the proxy.Dependencies and setupOn macOS the proxy's setup and installation instructions should automatically bundle or install all required dependencies.
Any error messages you may encounter (for example, with yourpipversion andcryptography, orpillowandimagingftdependencies, ormacOS SSL failures) normally give clear explanations of the issues and point to instructions for resolving these problems.
Pleaseopen an issueif you encounter any other problems here.When first launching on Linux in GUI mode you may encounter errors similar toNamespace […] not available, issues with the task bar icon display, or no browser popup when attempting to authorise your accounts.
This is caused by missing dependencies forpystrayandpywebview, which are used to display the menu bar icon and authentication windows.
See thepywebview dependenciesandpystray FAQpages andexistingclosed issuesin this repository for a summary and suggestions about how to resolve this.A similar issue may occur on Windows with thepythonnetpackage, which is required bypywebview.
If you are unable to resolve this by following thepythonnet installation instructions, you may find that installing aprebuilt wheelhelps fix the issue.
Note that the public releases of pythonnet can take some time to be compatible with the latest major python release, so it can be worth using a slightly older version of python, or a pre-release version of pythonnet.Known issuesWith some combinations of operating systems, web engines and virtual environments, keyboard control or input to the proxy's popup authorisation window may not always work.
On Windows this is normally limited to keyboard shortcuts (i.e., copy/paste), but in some virtual environments on macOS the entire keyboard may not work.
As a workaround, the proxy will enable pywebview's debug mode when you run the proxy itself in debug mode, which should allow you to use the right-click context menu to copy/paste to enter text.
If you are unable to proceed with popup-based authentication even with this workaround, it is worth trying the proxy's--external-author--local-server-authoptions.On more recent macOS versions (10.14 and later), you may find that when first running the proxy as a service you need to manually load its launch agent in order to trigger a file access permission prompt.
You will know intervention is necessary if the proxy exits (rather than restarts) the first time you clickStart at loginfrom its menu bar icon.
To resolve this, exit the proxy and then runlaunchctl load ~/Library/LaunchAgents/ac.robinson.email-oauth2-proxy.plistfrom a terminal.
A permission pop-up should appear requesting file access for python.
Once this has been approved, the proxy's menu bar icon will appear as normal.
In some cases — particularly when running the proxy in a virtual environment, or using the built-in macOS python, rather than the python.org version, or installations managed by, e.g., homebrew, pyenv, etc. — the permission prompt does not appear.
If this happens it is worth first trying tounloadand thenloadthe service vialaunchctl.
If this still does not cause the prompt to appear, the only currently-known resolution is to run the proxy outside of a virtual environment and manually grant Full Disk Access to your python executable via the privacy settings in the macOS System Preferences.
You may also need to edit the proxy's launch agent plist file, which is found at the location given in the command above, to set the path to your python executable – it must be the real path rather than a symlink (thereadlinkcommand can help here).
Fortunately this is a one-time fix, and once the proxy loads successfully via this method you will not need to adjust its startup configuration again (except perhaps when upgrading to a newer major macOS version, in which case just repeat the procedure).Other problemsPlease feel free toopen an issuereporting any bugs you find, orsubmit a pull requestto help improve this tool.Advanced featuresTheplugins varianthas an additional feature that enables the use of separate scripts to modify IMAP/POP/SMTP commands when they are received from the client or server before passing through to the other side of the connection.
This allows a wide range of additional capabilities or triggers to be added the proxy.
For example, theIMAPIgnoreSentMessageUpload pluginintercepts any client commands to add emails to the IMAP sent messages mailbox, which resolves message duplication issues for servers that automatically do this when emails are received via SMTP (e.g., Office 365, Gmail, etc.).
TheIMAPCleanO365ATPLinks pluginrestores "Safe Links" modified by Microsoft Defender for Office 365 to their original URLs.
TheSMTPBlackHole plugingives the impression emails are being sent but actually silently discards them, which is useful for testing email sending tools.
See thedocumentation and examplesfor further details, additional sample plugins and setup instructions.Potential improvements (pull requests welcome)Full feature parity on different platforms (e.g., live menu updating; monitoring network status; clickable notifications)Switch to asyncio? (with Python 3.12,PEP 594removed the asyncore package that the proxy is built upon – currently mitigated by the use ofpyasyncore)STARTTLS for IMAP/POP?Related projects and alternativesMichael Stepner has created aTerraform configurationthat helps run this proxy on a lightweight cloud server (AWS EC2).
Thiago Macieira has provided amakefile and systemd configuration files.
For Docker, blacktirion has anexample configuration.If you already use postfix, thesasl-xoauth2plugin is probably a better solution than running this proxy.
Similarly, if you use an application that is able to handle OAuth 2.0 tokens but just cannot retrieve them itself, thenpizauth,mailctloroauth-helper-office-365may be more appropriate.
There are also dedicated helpers available for specific applications (e.g.,mutt_oauth2), and several open-source email clients that support OAuth 2.0 natively (e.g.,Thunderbird,Mailspring,FairEmail,Evolution, etc.).DavMailis an alternative to this proxy that takes the same approach of providing a local IMAP/POP/SMTP server (and more) for Exchange/Office 365, though it does this by translating these protocols into Exchange API calls rather than proxying the connection.
That approach is very useful in situations where server-side IMAP/POP/SMTP is not supported or enabled, or the full Exchange capabilities are needed, but it has limitations in terms of speed and the number of email messages that can be retrieved.
This proxy was developed to work around these limitations for providers that do support IMAP/POP/SMTP natively.LicenseApache 2.0