LivePatch

= Design =
Xen 4.7 and later implements v1 of the Live Patch Design.
There are three parts to utilize this technology:
* Hypervisor implementing the LIVEPATCH_SYSCTL hypercall
* Tool to upload/apply/revert the payloads: xen-livepatch
* Tool to generate the payloads.
The first two are part of the Xen Project source:
 git://xenbits.xen.org/xen.git
while the last one is in a seperate repository:
 git://xenbits.xen.org/livepatch-build-tools.git
== How to enable it ==
The code for hypervisor is enabled by default (for x86). When compiling the hypervisor one must
change the config to build it.
It can be done by editing xen/.config to have CONFIG_LIVEPATCH=y or using:
 $ git clone git://xenbits.xen.org/xen.git
 $ cd xen/xen
 $ make menuconfig
And selecting Common Features|Live Patch live patching support.
It is marked as SUPPORTED for x86. For ARM 32&64 it is TECH PREVIEW.
Once you have it built and booted, you can verify that Live Patch is built in via:
 #xl dmesg | grep livepatch
 (XEN) livepatch: : build-id: 34c63348019acbe8bab69808a6f8ad7618fa7c12
(which you will only see if you build the hypervisor with debug=y)
or alternatively via:
 # xl info | grep build_id
 build_id : 34c63348019acbe8bab69808a6f8ad7618fa7c12
The value: 34c63348019acbe8bab69808a6f8ad7618fa7c12 is different on every build.
(This build-id is used to keep track of payloads dependencies and make sure they are built
against the correct hypervisor built).
The tool to patch/load/etc is called xen-livepatch and is also part of the Xen source tree.
It is used to load/apply/revert/unload the payloads.
== ELF payload file ==
The design details how the structures look, see Design of payload format for details.
The external tool:
 git://xenbits.xen.org/livepatch-build-tools.git
can generate the payloads against the hypervisor.
= Build hot-patches =
To build hot-patches aka payloads aka binary fixes against the hypervisor, you need
two things:
* Xen hypervisor which supports Live Patch hypercalls.
* Tool to build binary fixes against hypervisor.
The tool is available at at:
 git://xenbits.xen.org/livepatch-build-tools.git 
Or alternatively the three examples that are part of the Xen code base.
== How to build built-in examples ==
The Xen Project also includes three (four in Xen 4.8) regression test-cases that can be built
against the hypervisor which modify the xen_extra_version. This provides
an simple way to verify the changes as:
 # xl info | grep extra_version
will verify that the patch has taken place.
There are three regression/test-cases:
* xen_hello_world: changes xen_extra_version() to return "Hello World".
* xen_bye_world: over-writes the xen_hello_world patch so it returns "Bye World."
* xen_replace_world: Used to replace the two other payloads and make xen_extra_version() print "Hello Again World".
* xen_nop: Used to NOP xen_minor_version - which results in it reporting Xen 4.4
To build them:
 $ cd xen
 $ make -C xen tests
 ...
 $ find . -name *.livepatch
 ./xen/arch/x86/test/xen_replace_world.livepatch
 ./xen/arch/x86/test/xen_hello_world.livepatch
 ./xen/arch/x86/test/xen_bye_world.livepatch
 ./dist/install/usr/lib/debug/xen_replace_world.livepatch
 ./dist/install/usr/lib/debug/xen_hello_world.livepatch
 ./dist/install/usr/lib/debug/xen_bye_world.livepatch
NOTEIn Xen 4.8 and later versions they are in xen/test/livepatch directory.
NOTE Since they are built on the same source and binary as the hypervisor that is running - it automatically extracts the build-id and embeds the build-id.
You can use an simple Perl script located at livepatch_test.pl
to perform a variety of test-cases to see whether the livepatch functionlity works.
The idea is to do (on host with Xen hypervisor that has Live Patch built-in):
 $ # copy them the host.
 $ cd /usr/lib/debug
 $ ls
 xen-syms-4.7-unstable xen_bye_world.livepatch xen_hello_world.livepatch 
 xen_replace_world.livepatch 
 $ sudo /root/livepatch_test.pl
The tool should return an error code zero.
OLD INSTRUCTIONS
The idea is to do (on host with Xen hypervisor that has Live Patch built-in):
 $ # copy them the host.
 # cd /usr/lib/debug
 # ls
 xen-syms-4.7-unstable xen_bye_world.livepatch xen_hello_world.livepatch 
 xen_replace_world.livepatch 
 # xen-livepatch load xen_hello_world.livepatch 
 Uploading xen_hello_world.livepatch (12336 bytes)
 Performing apply:. completed
 # xl info | grep extra
 xen_extra : Hello World
 # xen-livepatch load xen_bye_world.livepatch 
 Uploading xen_bye_world.livepatch (9244 bytes)
 Performing apply:. completed
 # xl info | grep extra
 xen_extra : Bye World!
 # xen-livepatch upload replace_bye_and_hello xen_replace_world.livepatch 
 Uploading xen_replace_world.livepatch (9328 bytes)
 # xen-livepatch list
 ID | status
 ----------------------------------------+------------
 xen_hello_world | APPLIED
 xen_bye_world | APPLIED
 replace_bye_and_hello | CHECKED
 # xen-livepatch replace replace_bye_and_hello
 Performing replace:. completed
 # xl info | grep extra
 xen_extra : Hello Again World!
 # xen-livepatch list
 ID | status
 ----------------------------------------+------------
 xen_hello_world | CHECKED
 xen_bye_world | CHECKED
 replace_bye_and_hello | APPLIED
== livepatch-build-tools ==
The workflow of the tool is to:
* Build a Xen hypervisor without the patch (alternatively one can just point it to xen-syms which is preferred.
* Build a Xen hypervisor with a patch.
* Compare the two above.
* Create an payload with the difference.
The tool is not capable of:
* Generating payloads only with .data changes (as in only for variables),
* Inline patching - it only patches the whole function,
* Cannot NOP out functions.
This explanation was lifted from this link and modified a bit.
This transcript should demonstrate how to use the tool. We will create three patches:
* a) First depends on the hypervisor
* b) Second depends on the first one being applied (cumulative)
* c) Third one replaces the first a) and second one b).
NOTEThe a) and c) payloads need to embedded an build-id of the hypervisor it will be loaded against.
That means one SHOULD NOT recompile the source tree of the Xen hypervisor that is running.
Hence the workload here is done on a copy of the source. The b) depends on the build-id of the a) payload.
This assumes that the xen-syms built originally is in ~/xen-orig/xen/xen-syms
We use /tmp to store a copy of the source. The version of the source code
MUST match the version that /xen-orig/xen/xen-syms was built.
One can easily verify that by (on target system):
 # xl info | grep xen_changeset
 $ Fri Apr 15 10:35:21 2016 -0400 git:7552904
The build-id in this example is 34c63348019acbe8bab69808a6f8ad7618fa7c12 but it will
be different on each build. One can extract this by:
 # xl info | grep build_id
 build_id : 34c63348019acbe8bab69808a6f8ad7618fa7c12
 $ cd /tmp/
 $ git clone git://xenbits.xen.org/xen.git
 $ # Or alternatively one can do git clone ~/xen-orig
 $ cd xen
 $ git checkout 7552904
 
 $ # Copy .config from the built hypervisor.
 $ cp ~/xen-orig/xen/.config /tmp/config
 $ Make sure LIVEPATCH is enabled 
 $ cat /tmp/config | grep LIVEPATCH
 CONFIG_LIVEPATCH=y
 $ # Write a patch
 $ mkdir ~/work
 $ git diff > ~/work/test1.patch
 $ git reset --hard
 $ # Write another patch
 $ git diff > ~/work/test2.patch
 $ git reset --hard
 $ # Write another patch
 $ git diff > ~/work/test3.patch
 $ git reset --hard
 $ cd /tmp
 $ git clone git://xenbits.xen.org/livepatch-build-tools.git
 $ cd livepatch-build-tools
 $ make
 $ ./livepatch-build -s /tmp/xen -c /tmp/config -p ~/work/test1.patch -o out1 -d --depends 34c63348019acbe8bab69808a6f8ad7618fa7c12
 
NOTE Don't blindly plug in the build-id value! Use the one your hypervisor is built with.
NOTE If you are using a pre-release version also use .
Or alternatively make it automatic if you do this often:
 --depends $(readelf -Wn ~/xen-orig/xen/xen-syms | tail -1 | sed 's/.*Build ID: //')
The first Live Patch payload depends on the hypervisor build-id, while the second
one (test2) will depend on the first one (test1).
Extract the build-id of the first payload:
 $ readelf -Wn out1/test1.livepatch| tail -1 | sed 's/.*Build ID: //'
 dbbd14b6aa110b9065e2667d42ef4291eed6b821
NOTE The build-id varies per build. This value is just an example - yours will differ.
NOTE We use --xen-syms - for cumulative patches (one payload on top of another) it is imperative that we have the original. This is because there's no way that the hot patch can prelink against a function introduced by a new patch because it is placed in a random area of memory.
For this, you need to pass the xen-syms that is actually being used, so all the addresses when resolving symbols are precisely correct:
 $ ./livepatch-build -s /tmp/xen -p ~/work/test2.patch -o out2 -c /tmp/config -d --xen-syms ~/xen-orig/xen/xen-syms --depends dbbd14b6aa110b9065e2667d42ef4291eed6b821
And the last one - build against the hypervisor:
 $ ./livepatch-build -s /tmp/xen -p ~/work/test3.patch -o out3 -c /tmp/config --debug --depends 34c63348019acbe8bab69808a6f8ad7618fa7c12
 $ # copy out*/test*.livepatch onto the host (or if locally skip this step)
 
 
 On the host:
 # xen-livepatch upload test1 test1.livepatch
 # xen-livepatch upload test2 test2.livepatch
 # xen-livepatch upload test3 test3.livepatch
 # xen-livepatch list
 ID | status
 ----------------------------------------+------------
 test1 | CHECKED
 test2 | CHECKED
 test3 | CHECKED
 # # verify the build-ids match
 # xl debug-keys x
 # xl dmesg | tail -15
 (XEN) name=test1 state=CHECKED(1) ffff82d080a09000 (.data=ffff82d080a0a000, .rodata=ffff82d080a0a000) using 2 pages.
 (XEN) do_xen_version patch ffff82d080115069(1930) with ffff82d080a09000 (1956)
 (XEN) compat_xen_version patch ffff82d08013faf5(1853) with ffff82d080a097c8 (1879)
 (XEN) build-id=dbbd14b6aa110b9065e2667d42ef4291eed6b821
 (XEN) depend-on=34c63348019acbe8bab69808a6f8ad7618fa7c12
 (XEN) name=test2 state=CHECKED(1) ffff82d080a0c000 (.data=ffff82d080a0d000, .rodata=ffff82d080a0d000) using 2 pages.
 (XEN) compat_xen_version patch ffff82d08013faf5(1853) with ffff82d080a0c000 (1887)
 (XEN) do_xen_version patch ffff82d080115069(1930) with ffff82d080a0c783 (1964)
 (XEN) build-id=50364b9dbdb57e15f57bac237fce7fdfc37a94bd
 (XEN) depend-on=dbbd14b6aa110b9065e2667d42ef4291eed6b821
 (XEN) name=test3 state=CHECKED(1) ffff82d080a0f000 (.data=ffff82d080a10000, .rodata=ffff82d080a10000) using 2 pages.
 (XEN) do_xen_version patch ffff82d080115069(1930) with ffff82d080a0f000 (1958)
 (XEN) compat_xen_version patch ffff82d08013faf5(1853) with ffff82d080a0f7ca (1881)
 (XEN) build-id=0f2d152953babf40b6635e2c80adcb3ced2b04d9
 (XEN) depend-on=34c63348019acbe8bab69808a6f8ad7618fa7c12
 # xen-livepatch apply test1
 # # Verify test1 is applied
 # xen-livepatch apply test2
 # # Verify test2 is also applied
 # xen-livepatch list
 ID | status
 ----------------------------------------+------------
 test1 | APPLIED
 test2 | APPLIED
 test3 | CHECKED
 # xen-livepatch replace test3
 # # Verify test3 is applied and test1 and test2 are not
 # xen-livepatch revert test3
 # # Verify test3 is not applied
 
 # xen-livepatch unload test1
 # xen-livepatch unload test2
 # xen-livepatch unload test3
= TODOs =
v1 of patchset is in the Xen code base. We still have a list of TODOs and track it here.
== OSSTest ==
Make OSSTest do regression tests. Xen 4.9
== Toolstack ==
Things that need to be done in there:
* xen-livepatch better smarts. Right now you need to do 'upload','check' and 'apply'. Three operations which could be folded in one - since the system admin probably want to have all of those done at once. The 'all' option does that.
* Instead of using 'xen-livepatch' we should use 'xl'. That above mentioned logic could be part of 'xl' while 'xen-livepatch' is an accessory function for expert users?
* If hypervisor is compiled without Live Patch the tool should print simply '"livepatch support not available in Xen'
* If the build-id is wrong and one tries to apply the payload the message of 'hello failed with 1(Operation not permitted)' is lacking. We should have more details (and maybe a different error?). Or perhaps check the build-d as as being upload (however we may load them out of order). Perhaps have the toolstack check the build-id? How to do that against loaded patches (have them locally saved?)
* Read build-id of the payloads files and display them - as well the dependencies. Perhaps even add a simple dependency checker.
== livepatch-build--tools ==
* Provide better error-code - saying what is unsupported if the user tries to do it.
* Able to generate payloads that NOP functions.
* Able to generate payloads only against .data sections.
* Create payloads for ARM 32
* Create payloads for ARM 64
== Hypervisor ==
Currently the implementation has many TODOs implemented.
However we need to expand the hypervisor to include support for:
* build_id support Done! (4.7)
* The check for --build-id needs to deal with localized binutils. (4.7)
* Make it work with .rodata changes only, or just inline changes. (4.8)
* Need 'dl_sym' functionality in hypervisor to parse ELF payload. Do dynamic linking. (Ross) (4.7)
* Lookup and insertion in the symbol table. Compute the proper offset and virtual address based on symbol name (<do_domctl>) (Ross) (4.7)
* Need to either remove duplicate names from the symbol table or have the hypervisor implement proper ELF parsing - integrate the elf sections and parse that during runtime. (4.7)
* Insertion of exceptions in the exception table. Also support resorting it. The hot-patch may have new exceptions that need to be dealt with. (4.7)
* Checking the build-id against the hypervisor. (4.7)
* Code to patch and revert the hot-patch (for simplicity we could boot Xen with 'cpus=1' so we don't have to worry about proper code to trigger patching) (4.7)
** Support NMI? (or something similar?) (4.7)
** Patching requires saving the initial state. That needs to be saved for revert. (4.7)
** Patching needs allocation of memory, putting in the code at memory. (4.7)
** Deal with proper page table types for the modification (ro for .data, x for .text, etc). (4.7)
** Code to revert. (4.7)
* Code to activate patching mechanism. On VMEXIT routine right before checking the do_softirq. Ditto for PV paths. Call the top patching mechanism - which can IPI other CPUs (or wait until all CPUs have come to the same point). (4.7)
* Add functionality to execute ELF code. (4.8)
* Patch with NOPs if .livepatch.funcs->new_size has 0. (4.8)
* Seperate pages for .text; .rodata; .bss and .data (4.8)
* Make it work under ARM32:Done! (4.8)
** Support ARM32 ELF relocations
** Support ARM32 virtual addresses
** Support a similar to CPU flushing semantics as x86
** Support NMI? (or something similar?)
* Make it work under ARM64: (4.8)
** Support ARM64 ELF relocations
** Support ARM64 virtual addresses
** Support a similar to CPU flushing semantics as x86 
* A better mechanism to "mask" NMIs during patching. The existing mechanism looses NMI if they have been sent and we don't have a mechanism to replay them. Note that this is also fixes alternative section patching. Could (like Linux) annotate handlers don't get patched.
* Code to verify signature of the ELF payload.
** Doing the verification - checking if any code that uses the to-be-patched-function is in use. Save what the virtual address for the functions is - and verify the stack when patching (this incidentally is what .livepatch_section structures can have).
* tboot - will need to remeasure hypervisor when adding new code
* SecureBoot - not sure how it works with Live Patch. Needs to be tested. Xen should verify payload - and in turn use SecureBoot services to verify?
* Make .livepatch.funcs be an SHF_MERGE type with .sh_entsize having the size of the struct.
* Make the usage of tools/symbols use common list model (similar to how alternative code is built)
* When figuring out the new_addr and the name is <symbol>+<offset> - compute that.
* The code for doing the patching - call from __do_softirq - can also be used for runtime microcode loading.
* Make XENPF_get_symbol also include Live Patch symbols.
Category:Security
Category:LivePatch
Category:Xen 4.7