nyuuzyou/wikis huggingface dataset:
Compile the XAPI toolstack from source

There are 2 reasonably simple ways to build XAPI and related components:
== via opam ==
Opam is a source-based package manager which has a developer-friendly workflow. The following commands assume an Ubuntu host, update to match your distro of choice:
apt-get install opam -y
Next the dependencies of xapi can be installed:
sudo apt-get install $(opam install xapi -e ubuntu)
Finally xapi itself can be built:
opam install xapi
== via RPM ==
The open-source XenServer "buildroot" has a convenient Makefile to build Xapi via RPM (and experimentally via .debs also):
On a CentOS 6 machine:
yum install -y epel-release
After adding EPEL, install and set up mock:
yum install -y mock redhat-lsb-core
Mock will refuse to run as root. You must choose a non-privileged user to run mock as. Type the following as root:
(Note select a <user> which isn't "mock" when typing the commands below)
useradd <user> -G mock
passwd <user>
su - <user>
You are now ready to clone the buildroot repository and build the packages:
git clone git://github.com/xenserver/buildroot.git
cd buildroot
./configure
make
Category:XAPI Devel
Category:Development Process
Compressing VM Exports

Note: this has been implemented and is documented in the XenAPI reference
= Compressing VM exports =
== Background ==
There are two kinds of VM exports in XCP:
# metadata only
# full VM: metadata plus disk blocks
An export is a tar format archive where the metadata is always the first file and has name "ova.xml".
The metadata is stored in XML format and includes a bunch of existing version numbers:
key
product_version
build_number
xapi_major
xapi_minor
export_vsn
The CLI commands involved are:
 xe vm-export vm=... filename=...
 xe vm-import filename=...
Exports and imports can also be processed by HTTP GET and PUT.
== Problem statement ==
The "full VM" exports can be very large since they contain raw disk blocks. Their size makes them difficult to store and distribute over the network.
== Proposal ==
The disk blocks in a "full VM" export often contain very little user data and compress very well with gzip. I propose to:
# add a --compress option to the CLI which will cause VM exports to be compressed via gzip
# on import, auto-detect whether a VM requires decompression and decompress it first.
=== Auto-detection algorithm ===
Approximately:
# Read the first 512 bytes (a tar header length)
# Check if the tar header is well-formed and refers to an ova.xml file
## f so: consider as uncompressed
## If not: open a pipe to zcat, in a background thread write the first 512 bytes and then the rest of the file. Read from the output of zcat and look for a tar header
=== Version number discussion ===
The export format has two layers:
# the outermost ("envelope"?) layer which currently contains: tar with an xml file containing version numbers
# the innermost ("payload"?) layer which currently contains: the rest of the VM metadata and optional disk blocks
The changes anticipated in the future all concern the "innermost" layer and hence are covered by the existing version numbers. For example I anticipate
# adding new VM metadata
# changing the disk block format e.g. to vhd
Adding another version number to cover the outermost layer seems like overkill because:
# it is simple enough to tell the difference between tar and tar.gz automatically
# no future changes to this layer are anticipated
# if we change the outermost layer then we'll simply extend the auto-detect code.
== Implications ==
Old (uncompressed) exports will continue to import on new servers, thanks to the auto-detection.
New compressed exports will not import on old hosts. They will fail with a generic "IMPORT_ERROR" exception (probably containing the text 'Failure "int_of_string"'). A simple workaround is to uncompress the file first and then import it.
Category:XAPI Devel
ConVirt

From the ConVirt documentation:
"ConVirt is a powerful virtualization management platform for administering and automating your Xen infrastructure. Use ConVirt if you need centralized management; if you need high availability, backup & restore, and workload management; if you need self-service for your customers; or if you want to manage your virtualization infrastructure from a "single pane of glass"."
== STATUS ==
* ConVirt 2.5 is Open Source. It can be installed locally or as a pre-built VM image from the Convirture website.
* ConVirt currently integrates with xm/xend. Support for xl/libxenlight and/or libvirt is on their roadmap.
* Integration logic is a little stale, apparently with the last explicitly supported version of Xen Project software being 3.2. You may need to manually do a few tasks to integrate with your 4.* installation.
== DOCUMENTATION ==
* ConVirt for Xen Project write-up and Webinar from the Convirture website
Category:Integration
Category:GUI
Connecting a Console to DomU's

This article describes the common guest console operations.
= Connect to guest console =
From dom0:
<nowiki>
# xm console <domid>
</nowiki>
It will invoke /usr/lib/xen/bin/xenconsole.
Instead, you can use this shell xenconsole.sh (require socat):
<nowiki>
# #!/bin/sh
domid=$1
tty=$(xenstore-read /local/domain/${domid}/console/tty)
socat - ${tty},IGNBRK=0,BRKINT=0,PARMRK=0,ISTRIP=0,INLCR=0,IGNCR=0,ICRNL=0,IXON=0,OPOST=0,ECHO=0,ECHONL=0,ICANON=0,ISIG=0,IEXTEN=0,CSIZE=0,PARENB=0,CS8
</nowiki>
Invoke it by:
<nowiki>
# xenconsole.sh <domid>
</nowiki>
= Connect two guest consoles =
Some times useful to connect two guest consoles, eg. debugging a guest from another by emulated serial console.
You can use this shell connect.sh (require socat):
<nowiki>
#!/bin/sh
src=$1
dst=$2
srctty=$(xenstore-read /local/domain/${src}/console/tty)
dsttty=$(xenstore-read /local/domain/${dst}/console/tty)
socat ${srctty} ${dsttty}
</nowiki>
Invoke it by:
<nowiki>
# connect.sh <domid1> <domid2> 
</nowiki>
= Reference =
* socat - Multipurpose relay.
* http://os-drive.com/wiki/XenDomainConsole
Category:Xen
Category:HowTo
Category:Users
Category:Beginners
Contributor Training

== Part 1: Introduction (mostly generic) ==
* 
* Goal: Effective Contributions
* Covers: Understanding stakeholders, the basic of code reviews, common mistakes, planning code submissions
== Part 2: Processes and Conventions (Xen Project specific) ==
* XP Contributor Training - Pt 2 Processes and Conventions v1.1.pdf - split into two files due to file size
** 
** 
* Goal: Highlights Xen Project specific Processes and Conventions
* Covers: Xen Project specific roles, processes and conventions
== Part 3: Communications (generic) ==
* XP Contributor Training - Pt 3 Communication v1.0.pdf - split into two files due to file size
** 
** 
* Goal: Effective Communication techniques to come up with the best compromise
* Covers: A host of different communications techniques
== Part 4: Culture (Xen Project specific) ==
* XP Contributor Training - Pt 4b Culture.pdf - split into three files due to file size
** 
** 
** 
* Goal: Gives a sense of the culture within the project
* Covers: Overview of past and ongoing improvement initiatives within the Xen Project and what they say abut the culture in the project
Creating a LVM backed XFS SR

LVM-backed XFS SR
----
These instructions were fist tested on a Kubuntu 11.10 system, but should work with any Ubuntu/Debian Kronos-based system to create an LVM-backed XFS SR
The following is what was required:
Install lvm2 and xfstools.
apt-get install lvm2 xfstools
Create a logical volume
 
(If you need help in doing this, there are many tutorials for LVM on the net)
( see )
Then you must copy the XFSSR.py
(Get XFSSR.py here)
cp XFSSR.py /usr/lib/xcp/sm/
Create a symlink to XFSSR
cd /usr/lib/xcp/sm/
ln -s XFSSR.py XFSSR
Once the logical volume is created, restart xapi
service xcp-xapi restart
To create the SR, run
SR=`xe sr-create type=xfs device-config:device=/dev/mapper/test-builder name-label=xfs`
Change '/dev/mapper/test-builder' to your lvm device to match your system
After that is done, you'll be able to use the XFS LVM filesystem as the default SR
Enjoy!
CrossPoolMigration

This page describes current work to make
* Cross-pool migration
* Storage migration
work on XCP
The following designs were considered:
# CrossPoolMigrationv1: based on DRBD
# CrossPoolMigrationv2: based on tapdisk log-dirty
# CrossPoolMigrationv3: based on tapdisk mirroring
= Choosing a design =
The following properties are desired:
# the chosen design should exploit all available disk structure information to minimise bandwidth and be fast
# the chosen design should clearly separate the storage migration from the domain migration, avoiding the need to add additional hooks into the domain memory send/receive code (in libxenguest)
#* this should also make it easier to use libxl functions in future
# the chosen design should be as "live" as possible i.e. it should avoid extending the migration downtime
# the chosen design should be compatible with *dom0 disaggregation*, in particular where some storage elements are not in dom0
= Proposed choice =
The design based on tapdisk mirroring CrossPoolMigrationv3 is the proposed choice. It has the following benefits:
# it allows vhd-based storage backends to copy only disk blocks that cannot be found on the destination SR
## furthermore if a migration fails, many of the blocks will probably have been copied so subsequent migrations may be faster
# it separates "mirror creation" from "domain migration", so the libxenguest is unmodified
# by creating a mirror in advance, the code running in the "migration downtime" is unmodified
# *unknown*: the ease of disaggregating dom0 will depend on the exact APIs used, these are still TBD
Category:XAPI Devel
CrossPoolMigrationv1

This page describes a possible design for Cross-pool migration (which also works for within-pool migration with and without shared storage).
This is an alternative design to CrossPoolMigrationv2 and CrossPoolMigrationv3
This design has the following features:
# DRBD is used to replicate disks on demand
#* this is storage-format agnostic: it doesn't require .vhd
# the SM on one host can generate URIs which can be used by other SMs to bootstrap the disk mirroring process
# a single codepath is used in xapi, making testing easier
#* this replaces the existing migration API
# a simple RESTful API makes the whole thing quite easy to test, prod etc.
# the xapi pieces and the SM pieces can be developed and tested independently and integrated at the end
= Pros/Cons of DRBD vs snapshot/copy =
* Pro: by separating storage replication from memory transfer we can switch easily to "libxl" without modifications. Otherwise we would need callbacks from libxl to synchronise the disk copy with the memory copy.
* Pro: we don't need to write a snapshot/copy loop
* Con: we do need to write something to manage a drbd instance, which may be on a shared host (eg vanilla Debian dom0)
* Pro: migration downtime is expected to be lower, since replication cost is spread over each I/O request, rather than being bursty
* Con: continuous replication might slow down some workloads more than snapshot/copy
* Con: DRBD is linux only: it's not clear how this would work with (eg) a FreeBSD storage driver domain.
= Component diagram =
Component PDF file
= Migration sequence =
Migrate PDF file
= Proposed milestones and task list =
The following milestones are proposed:
Milestone number
1
2
3
4
The following tasks are proposed:
Task 	Status
Create sample drbd.conf files 	Completed
Implement a DRBD service in dom0 	In progress
Implement xapi HTTP operations 	
Implement SMAPI VDI.replicate_to, VDI.get_replication_target	
Implement xapi HTTP heartbeat 	
Implement XenAPI VM.migrate, VM.receive 	
Category:XAPI Devel
CrossPoolMigrationv2

This page describes a possible design for Cross-pool migration (which also works for within-pool migration with and without shared storage).
This is an alternative design to CrossPoolMigrationv1 and CrossPoolMigrationv3.
This design has the following features:
# tapdisk log-dirty is used to build an efficient continuous disk copy program
#* this is storage-format agnostic: it doesn't require .vhd
# the SM on one host can generate URIs which can be used by other SMs to access disks over iSCSI on the storage network
# handling of iSCSI endpoints (both initiators and targets) is abstracted by an iSCSI service
#* this is expected to simplify the current code and prevent (eg) a bug in one SM plugin from interfering with another SM plugin
# a single codepath is used in xapi, making testing easier
#* this replaces the existing migration API
# a simple RESTful API makes the whole thing quite easy to test, prod etc.
# the xapi pieces and the SM pieces can be developed and tested independently and integrated at the end
= Component diagram =
Components (PDF)
= Migration sequence =
Migrate (PDF)
= Proposed milestones and task list =
The following milestones are proposed:
Milestone number
1
2
3
4
The following tasks are proposed:
Task 	Status
Forward-port blktap1 log dirty mode to tapdisk2 	
Implement an iSCSI service in dom0 	Completed
Modify VDI.deactivate to finish any pending copies	
Implement xapi HTTP operations 	
Create a continuous copy program (use sparse_dd?) 	
Implement SMAPI VDI.copy 	
Implement SMAPI VDI.get_url 	
Implement XenAPI VM.migrate, VM.receive 	
Implement xapi HTTP heartbeat 	
= Links =
The iscsi-target-manager is a simple python program which wraps the "tgtadm" commands used to configure an iSCSI target.
Category:XAPI Devel
Debugging QEMU

There are rare times when debugging problems in the Xen Project software may mean you will want to run QEMU under a debugger.
To run gdb on qemu-dm, replace the /usr/lib/xen/bin/qemu-dm by a script:
 #!/bin/sh
 exec gdbserver 0.0.0.0:1234 /usr/lib/xen/bin/qemu-dm.bak $@
And run gdb. `target remote localhost 1234` to connect to gdbserver.
With recent versions of Xen Project software (4.1 and following), you can specify a different device model in the config file instead of replacing the default binary.
== REFERENCES ==
* https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg00302.html
* https://lists.gnu.org/archive/html/qemu-devel/2011-12/msg02575.html
Category:Developers
Design Sessions 2017

== 2017: Developer and Design Summit ==
=== Notes ===
Intel sessions
* Notes from the 5-level-paging session 
ARM Sessions
* Notes from PCI Passthrough design discussion at Xen Summit
Coprocessor Sharing and Stubdoms
* Notes on stubdoms and latency on ARM
* Shared coprocessor framework followup
* SCF configuration followup
PVH (v2)
* Notes from the PVH toolstack interface session
* Notes from the PVH performance session
Release and Build tools
* Notes Design Session: Making Releases Lessons Learned: Improving Our Release Process and Tooling
* Notes Design Session: Testing & CI Process and Workflow Improvements, x86/ARM/Embedded Testing, etc. - Does what we do today work?
Outcomes
* Build tools follow up
* A document for Xen release management 
* A document for Xen release management, v2
Security, Safety
* Notes of Design Session: Xen Certification in Automotive Industrial
* Notes for Design Session: Loose ends for becoming a CNA (CVE Numbering Authorities) and other Security Team Operational Questions 
* Notes from Design Summit Hypervisor Fuzzing Session 
Outcomes
* Scripts to check XSA patch-level on xen trees (xen.git, qemu-xen.git & qemu-xen-traditional.git)
Community
* Notes from Design Session: Solving Community Problems: Patch Volume vs Review Bandwidth, Community Meetings ... and other problems
=== Sessions Highlighted on the Wiki ===
Category:Developers
Category:Design Sessions
Category:Events
Developer Meeting/Aug2014

== When and Where ? ==
August 20, 9:30-13:00
Room: Ontario
Sheraton Chicago
301 E North Water St
Chicago
IL 60611
United States
=== Break ===
11:00 - 11:30, Coffee, Tea and Snacks provided 
=== Hotel Options ===
See Xen Project Developer Summit
== Topics to Discuss ==
=== Instructions ===
Alternatively, you can reply to this mail and we will add you.
=== Topics ===
Please add your topic here
== Confirmed Attendees ==
* Lars Kurth, Citrix - can only do part of the meeting as my LinuxCon talk is at 12:15 - 13:00. Looking for somebody to chair/host the meeting. I am also willing to give up my space if need be.
* Don Slutz, Verizon
* Don Koch, Verizon (low priority: willing to give up my space)
* Daniel Kiper, Oracle
* Jim Fehlig, SUSE
* Chunyan Liu, SUSE
* Aravindh Puthiyaparambil, Cisco
* Jan Beulich, SUSE
* Luis R. Rodriguez, SUSE
* David Vrabel, Citrix
* George Dunlap, Citrix
* Anil Madhavapeddy, Cambridge University
* Julien Grall, Citrix
* Russell Pavlicek, Citrix (lowest priority -- bump me if you need my chair; like Lars, I have a talk at 12:15)
Category:Project
Category:Community
Category:Events
Developer Meeting XS13

== When and Where ? ==
October 23, 13:00-16:00
Melville Suite
2nd floor
Sheraton Grand
1 Festival Square
Edinburgh EH3 9SR
This is only a short walk from the Edinburgh International Conference Centre
Check out the map below (note that you can probably walk via Rutland Court at the back of the Conference Centre)
== Topics to Discuss at the Developer Meeting ==
=== Instructions ===
Please reply to this thread or update the wiki
=== Topics ===
* 4.4 schedule (Jan)
* release schedule plans going forward (and their effect on stable tree maintenance) (Jan)
* PVH situation (Jan)
* kexec situation (Jan)
* design docs and design review for significant hypervisor features (David Vrabel)
* Linux arch/ia64/xen status (Boris)
* EFI + GRUB2 (multiboot2 protocol) + Xen (Daniel, somebody else will need to lead as Daniel is not there in person)
== Confirmed Attendees ==
* Lars Kurth, Citrix - can only do part of the meeting as my LinuxCon talk is at 14:30 - 15:30 on that day. Looking for somebody to chair/host the meeting. I am also willing to give up my space if need be.
* Ian Campbell, Citrix
* Deep Debroy, Cisco
* George Dunlap, Citrix
* Roger Pau Monné, Citrix
* Matt Wilson, Amazon Web Services
* Dario Faggioli, Citrix
* Julien Grall, Citrix
* Ben Guthro, Citrix
* Tim Deegan
* Jan Beulich, SUSE
* David Vrabel, Citrix
* Anthony PERARD, Citrix
* Ian Jackson, Citrix
* Boris Ostrovsky, Oracle
* Joao Martins, NEC
* Wei Liu, Citrix
* Don Slutz, Verizon
* Harry Hart, Verizon
* Russell Pavlicek, Citrix
* Stefano Stabellini, Citrix
Category:Project
Category:Community
Category:Events
Device tree

== Specifications ==
* Power.org Standard for Embedded Power Architecture Platform Requirements (ePAPR) (sections 2, 3, 6 and 7).
== Xen Specific Requirements ==
Note: These are subject to change.
=== bootargs for dom0 ===
The bootargs string for dom0 shall be provided as the xen,dom0-bootargs parameter in the /chosen node.
When preparing the device tree for dom0, Xen will set bootargs to the xen,dom0-bootargs value in the original device tree.
Example:
 / {
 [...]
 
 chosen {
 bootargs = "console=serial0 dom0_mem=128M";
 xen,dom0-bootargs = "console=ttyAMA1 root=/dev/mmcblk0";
 }
 
 [...]
 }
Category:XenARM
Dom0

Dom0 is the initial domain started by the Xen hypervisor on boot. Dom0 is an abbrevation of "Domain 0" (sometimes written as "domain zero" or the "host domain"). Dom0 is a privileged domain that starts first and manages the DomU unprivileged domains. 
The Xen hypervisor is not usable without Dom0. This is essentially the "host" operating system (or a "service console", if you prefer). As a result, Dom0 runs the Xen management toolstack, and has special privileges, like being able to access the hardware directly. 
Dom0 has drivers for hardware, and it provides Xen virtual disks and network access for guests each referred to as a domU (unprivileged domains). For hardware that is made available to other domains, like network interfaces and disks, it will run the BackendDriver, which multiplexes and forwards to the hardware requests from the FrontendDriver in each DomU. 
Unless DriverDomain's are being used or the hardware is passed through to the domU, the dom0 is responsible for running all of the device drivers for the hardware. 
== Also See ==
* Xen Project Software Overview
* Dom0 Kernels for Xen : Supported Kernels for Dom0
Most commonly dom0 runs some variant of Linux, but Xen dom0 implementations also exist for OpenSolaris and NetBSD. Modified versions of Linux, NetBSD and Solaris can be used as the dom0. See OS vendor documentation for specific steps. Current official documentation may be found at the following sites:
* NetBSD -- http://www.netbsd.org/ports/xen/howto.html#netbsd-dom0
* OpenSolaris -- http://www.opensolaris.org/os/community/xen/docs/SetupDom0.htm
Category:Xen
Category:Users
Category:Beginners
Category:Overview
Dom0 Disaggregation

==Dom0 Disaggregation==
The Dom0 or control domain has traditionally been a monolithic privileged virtual machine. However, the original intention was for Dom0 to be carefully broken into several privileged service domains - termed Dom0 disaggregation. Qubes OS, OpenXT, Citrix XenClient, and Xoar have made the case for more disaggregation of Dom0 for purposes of better security, reliablity, isolation, and auditability.
==In Practice==
The OpenXT project, derived from the original Citrix product XenClient XT, offers a disaggregated networking stack and isolated network driver domains.
Qubes OS is similarly pursuing a disaggregated architecture for a desktop system.
The Citrix XenServer organization has made steps towards disaggregation, with the Windsor architecture project.
"Although domain 0 disaggregation is not new it is seldom used in practise and much of its use is focussed on providing enhanced security. Citrix XenServer will be moving towards a disaggregated domain 0 in order to provide better security, scalability, performance, reliability, supportability and flexibility."
Project “Windsor”: Domain 0 Disaggregation for XenServerXCP:
* http://www.xen.org/xensummit/xs12na_talks/T2.html
A more recent update indicates that the Windsor project is now viewed more as technology development than productization:
* http://xenserver.org/discuss-virtualization/virtualization-blog/entry/whatever-happened-to-xenserver-s-windsor-architecture.html
==References==
* http://openxt.org
* http://qubes-os.org/Home.html
* http://www.cs.ubc.ca/~andy/papers/xoar-sosp-final.pdf
* http://www.slideshare.net/cloudstack/vbacd-xen-cloud-computing
* http://vimeo.com/38636349
Category:Security
Category:Glossary
DomU

__NOTOC__
A DomU is the counterpart to Dom0; it is an unprivileged domain with (by default) no access to the hardware. It must run a FrontendDriver for multiplexed hardware it wishes to share with other domains. A DomU is started by running 
xl create <xen-config-filename>
in Dom0. The kernel for a DomU comes from Dom0's filesystem, not from the filesystem exported to the DomU.
== Also see ==
* DomU Support for Xen : List of supported operation systems
Category:Glossary
Domain

__NOTOC__
When we are talking about virtualization, a domain is one of the virtual machines that run on the system. Domain0 is the first domain started by the Xen hypervisor at boot, and will be running a Linux OS. This domain is privileged: it may access the hardware and can run the XenControlTools that manage other domains. These other domains are referred to as DomUs, the U standing for "user". They are unprivileged, and could be running any operating system that has been ported to Xen.
Category:Xen
Category:Glossary