Package: offlinebooks
offlinebooksKeep your accounts offline.Offlinebooks is a cmdline tool that downloads your Xero financial accounts for
local backup. It is beta quality software and only Linux (GNOME desktop) is
supported at present. Use with caution. The code repository is
atofflinebooks.We are not associated with the Xero organisation and this software is
unsupported. SeeLicensingfor more details.Table of ContentsInstallBackgroundUsageKnown IssuesRoadmapContributingLicensingAttributionInstallPrerequisites:Linux (GNOME desktop)Python 3XoauthInstall the latest version via pip:pipinstallofflinebooksThis will install a cmdline toolofflinebooks. For usage see below.BackgroundI just wanted a simple way to pull my data from the Xero API and save it locally
for backup.We save each entity as JSON in its own file, named by its unique ID (where
available) in a simple dir tree structure. This allows for easy processing with
other tools and is also suitable for source control. If an entity has attachments
those are saved next to the entity under a dir named[entity ID]_attachmentsData is saved separately for each tenant (organisation) under their tenant id
and under their human-readable tenant name as a symlink for convenience:$XDG_DATA_HOME/offlinebooks/
├── tenantName
│   ├── Demo Company (UK) -> $XDG_DATA_HOME/offlinebooks/tenantId/b3b892ur-02i8-4842-8bx2-85696h032kz2
│   └── ...
└── tenantId
    ├── b3b892ur-02i8-4842-8bx2-85696h032kz2
    │   ├── accounts
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7.json
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7_attachments
    │   │   │   └── ...
    │   │   └── ...
    │   ├── brandingthemes
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7.json
    │   │   └── ...
    │   ├── contacts
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7.json
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7_attachments
    │   │   │   └── ...
    │   │   └── ...
    │   ├── currencies
    │   │   ├── GBP.json
    │   │   └── ...
    │   ├── invoices
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7.json
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7_attachments
    │   │   │   └── ...
    │   │   └── ...
    │   ├── items
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7.json
    │   │   └── ...
    │   ├── journals
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7.json
    │   │   └── ...
    │   ├── organisations
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7.json
    │   │   └── ...
    │   ├── taxrates
    │   │   ├── 15% (VAT on Capital Purchases).json
    │   │   └── ...
    │   └── users
    │   │   ├── a6r01id3-690x-7edt-8pd2-c873245y38v7.json
    │   │   └── ...
    └── ...Data covered so far:AccountsBank TransactionsBank TransfersBranding themeContactsCurrenciesInvoicesItemsJournalsManual JournalsOrganisations, EXCEPT:Organisation actionsCIS Settings (UK)PaymentsUsersUsageThe first time around you will need to add offlinebooks as a PKCE app to your Xero
account and grant it permissions:Thanks toJWealthallyou can simply follow his excellentPKCE How To for Xero OAuth2
APIwith the following
changes:Useofflinebooksas both the 'App name' in Xero and the clientname when
runningxoauth setup.For company or application URL you can put whatever you like or can I suggest
theofflinebooks project page on PyPIWhen it gets to adding scopes we needopenidto authorise the app andoffline_accessto refresh our token on expiry. Beyond that
we just need read permissions for each entity we download. DO NOT ADD ANY
WRITE PERMISSIONS, they are not needed. The following should suffice:xoauthsetupadd-scopeofflinebooks\openid\offline_access\accounting.transactions.read\accounting.contacts.read\accounting.journals.read\accounting.settings.read\files.read\accounting.reports.read\accounting.attachments.readOnce you've completed the How To then you are ready to run offlinebooks:offlinebooksIf it completes without error you'll find a new dir at$XDG_DATA_HOME/offlinebooks(probably~/.local/share/offlinebooks)
containing the downloaded data for you to explore (see above).If it hits the Xero API minute rate-limit it'll pause for a time interval (as
supplied by the API) before continuing the run. You'll see some output to
indicate this, for example:Xero API rate-limit exceeded for calls per minute, will pause for 43.0s
API reported the following remaining limits:
  app minute : 9911
  daily      : 2718TroubleshootingNo such file or directory: '~/.xoauth/xoauth.json'- have you run
xoauth as detailed above?Known IssuesPossibly fixed since v0.0.8: still get API rate-limits exceeded sometimes:XeroAPIrate-limitexceededforcallsperminute,willpausefor14.0s
APIreportedthefollowingremaininglimits:appminute:9984daily:4794Traceback(mostrecentcalllast):File"/home/user/.local/bin/offlinebooks",line8,in<module>sys.exit(main())File"/home/user/.local/lib/python3.9/site-packages/offlinebooks/main.py",line329,inmainforentityinfetcher.fetch():File"/home/user/.local/lib/python3.9/site-packages/offlinebooks/main.py",line237,inall_generatoritems=func.all()File"/home/user/.local/lib/python3.9/site-packages/xero/basemanager.py",line264,inwrapperraiseXeroRateLimitExceeded(response,payload)xero.exceptions.XeroRateLimitExceeded:pleasewaitbeforeretryingthexeroapi,thelimitexceededis:minuteThis is hard to reproduce. I have added (24th Mar '22) an extra second to our
wait time in case of rounding issues though I have seen this exception occur
before we decide to wait. Also added extra logging in case we are reaching
our retry stop limit without realising.Intermittent exception 'oauthlib.oauth2.rfc6749.errors.InvalidGrantError' -
not sure what causes this. If you do axoauth connectthen run again it
should work.A long delay then a failure with exceptionxero.exceptions.XeroExceptionUnknown: Nonewhilst calling func.filter with
paging. If youxoauth connectthen run again it works.We depend onsecret-toolto retrieve the token whichxoauthhas saved,
this is GNOME-specific.Attachments are only re-fetched if they differ in size to avoid large
downloads. There's a small chance that an updated attachment which is the same
size as the original is not fetched.We do not yet download anything outside of the Accounting API, that leaves
these APIs untouched:PayrollAssetsFilesProjectsBank FeedsXero HQPractice ManagerWorkflowMaxWithin the Accounting API we do not yet save the following (for some we also
show in parenthesis where these are found in the Xero web GUI):Batch PaymentsBudgetsContact GroupsCredit NotesEmployeesHistory and NotesInvoice RemindersLinked TransactionsPayment ServicesPrepaymentsPurchase OrdersQuotesRepeating Invoices (Business-Invoices-Repeating)Reports (Accounting-Reports)Tracking Categories (Accounting-Advanced-Tracking Categories)RoadmapIn vague priority order:Download remaining data not yet supported from Accounting API above.How are VAT returns covered by the API?Allow user to limit tenant(s). Introduce config file in XDG friendly locationAllow user to specify repo path alsoWe should report at the end on API usage if requested: 'Each API response you
receive will include the X-DayLimit-Remaining, X-MinLimit-Remaining and
X-AppMinLimit-Remaining headers telling you the number of remaining against
each limit.' This can be got from the final response we receive.Fetch in parallelWe assume journals start at 1, i.e. setting offset=0 which means querying
JournalNumber>0. Is this definite?Our use of tenacity and handling API limits:We repeat the retry decorator on every function containing an API call.
Better to remove this duplication.It would be better not to reach into the response and get headers,
perhaps pyxero's XeroRateLimitExceeded exception could hold the data
instead.ContributingIt's great that you're interested in contributing. Please ask questions by
raising an issue, be sure to get in touch before raising PRs to avoid wasted
work. For full details seeCONTRIBUTING.mdLicensingWe declare our licensing by following the REUSE specification - copies of
applicable licenses are stored in the LICENSES directory. Here is a summary:All source code is licensed under AGPL-3.0-or-later.Anything else that is not executable, including the text when extracted from
code, is licensed under CC-BY-SA-4.0.For more accurate information, check individual files.Offlinebooks is free software: you can redistribute it and/or modify it under the
terms of the GNU Affero General Public License as published by the Free Software
Foundation, either version 3 of the License, or (at your option) any later
version.This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.Attribution"Xero" is a trademark of Xero Limited.