AFT installation

== A MinnowBoard MAX working system ==
=== The AFT elements and net ===
The testable devices are put in an isolated network (AKA AFT net). 
Line color code
Blue 	AC cable
Orange	USB cable
Green 	Ethernet cable
Black 	Not a cable
=== The high level testing flow ===
Once AFT was installed and configure. It'll load its configuration options from '/etc/aft/*.cfg' files and reserve a device by using a lockfile so multiple AFT instances can't use the same device at the same time. Then the device is turned off and then on using the Remote power switch that’s defined in the device config files to reboot the device. When we turn the device on we start to send keystrokes right away from the file determined by service_mode_keystrokes in the AFT configuration files. These keystrokes will go to BIOS and choose the support image to boot from. So for example with basic Minnowboard max as DUT we would send F2-key multiple times to make sure we get to BIOS, then use the arrow keys to go to boot menu and then choose the memory stick as the bootable device and then send Enter. After that we try to connect to the DUT with ssh using the ip in the dnsmasq.leases file. We use the 'id' from device config files to know what ip address to use. When we can connect to the DUT with ssh, we make sure that DUT has booted to 'service_mode' by using 'uname -a' command on the device. After that we flash the testable image and inject ssh-keys to it so we can access it easily when we boot it. When the flashing has been successful we reboot the device. The procedure is the same as before, but now utilizing test_mode_keystrokes for booting the testable OS and 'test_mode' to know we have really booted the right OS. After that we invoke the tests on the DUT and on default turn off the device after everything has completed.
=== Setup harness server ===
==== OS Installation ====
The Linux distribution installed upon harness server during the creation of this document was Ubuntu yakkety.
===== Important facts upon installation =====
During Installation process of Ubuntu It'll be requested the creation of a user, please create the user with the name tester. Additionally the computer name will have to be testing-harness.
When installation has finished. The 'sudo group' has to be enabled as passwordless upon file '/etc/sudoers'.
# Allow members of group sudo to execute any command
%sudo ALL=(ALL) NOPASSWD: ALL
It is important to denote that user tester has been set as part of 'sudo group' by the default Ubuntu installation.
With user tester is going to be executed the overall of automated testing.
===== Ram and Disk distribution =====
The Ram and Disk details this harness server during the creation of this document were:
root@testing-harness:~# free -m
 total used free shared buff/cache available
Mem: 15945 251 12904 54 2790 15311
Swap: 16278 0 16278
root@testing-harness:~# fdisk -l /dev/sda
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: B15E7DED-0607-4F32-8072-8CEC8041A1A8
Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 1920184319 1919133696 915.1G Linux filesystem
/dev/sda3 1920184320 1953523711 33339392 15.9G Linux swap
root@testing-harness:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.8G 0 7.8G 0% /dev
tmpfs 1.6G 9.3M 1.6G 1% /run
/dev/sda2 901G 39G 817G 5% /
tmpfs 7.8G 156K 7.8G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda1 511M 3.4M 508M 1% /boot/efi
tmpfs 1.6G 44K 1.6G 1% /run/user/111
tmpfs 1.6G 0 1.6G 0% /run/user/1000
==== Firewall configuration ====
Firewall configuration is not provided, it is recommended that the testing harness is only kept in a trusted network.
To check if no firewall are configured over testing-harness, we can expect a similar output when inquiring iptables... 
root@testing-harness:~# iptables -L && iptables -t nat -L
Chain INPUT (policy ACCEPT)
target prot opt source destination 
Chain FORWARD (policy ACCEPT)
target prot opt source destination 
Chain OUTPUT (policy ACCEPT)
target prot opt source destination 
Chain PREROUTING (policy ACCEPT)
target prot opt source destination 
Chain INPUT (policy ACCEPT)
target prot opt source destination 
Chain OUTPUT (policy ACCEPT)
target prot opt source destination 
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination 
==== Setup Security shell ====
OpenSSH is a freely available version of the Secure Shell (SSH) protocol family of tools for remotely controlling, or transferring files between, computers. That's why it is required its installation upon testing-harness server.
root@testing-harness:~# aptitude install openssh-server 
==== Setup NFS ====
AFT applies Network file system to share /home/tester directory where images and other files used in flashing for PC-devices within AFT net can be found.
===== NFS is installed ? =====
To check if the NFS server is installed, we can do...
root@testing-harness:~# dpkg -l | grep nfs-kernel-server
===== How to install NFS ? =====
To install NFS these are the required packages... 
root@testing-harness:~# aptitude install nfs-kernel-server
The following NEW packages will be installed:
 keyutils{a} libnfsidmap2{a} libtirpc1{a} nfs-common{a} nfs-kernel-server 
 rpcbind{a} 
0 packages upgraded, 6 newly installed, 0 to remove and 459 not upgraded.
Need to get 470 kB of archives. After unpacking 1 884 kB will be used.
Do you want to continue? [Y/n/?]
===== Sharing /home/tester directory =====
The tester’s home will be set as shared directory 
root@testing-harness:~# cd /etc
root@testing-harness:~# echo '/home/tester 192.168.1.0/24 (crossmnt,ro,root_squash,sync,no_subtree_check)' >> exports
==== Setup DNS cache and DHCP server for AFT net ====
DNSmasq provides two services, either of which can be used independently.
# DNS service
# DHCP service
A local DNS cache can speed up internet browsing because the user's applications will not need to access a domain name server when it looks up a domain name the computer has visited before.
===== How to install dnsmasq ? =====
To install dnsmasq these are the required packages... 
root@testing-harness:~# aptitude install dnsmasq
The following NEW packages will be installed:
 dns-root-data{a} dnsmasq dnsmasq-base{a} 
0 packages upgraded, 3 newly installed, 0 to remove and 459 not upgraded.
Need to get 0 B/329 kB of archives. After unpacking 886 kB will be used.
Do you want to continue? [Y/n/?
At the sametime we have set up a simple DNS server. To test it, use your favorite DNS lookup tool pointed at localhost
root@testing-harness:~# nslookup yoctoproject.org localhost
Server:		localhost
Address:	127.0.0.1#53
Non-authoritative answer:
Name:	yoctoproject.org
Address: 140.211.169.56
===== Configure DHCP server =====
To enable the DHCP server of AFT net, you will need to give it a range of IP addresses to hand out. The AFT net server would hand out 92 address starting at 192.168.1.2 and ending at 192.168.1.99.
root@testing-harness:~# echo 'dhcp-range=192.168.1.2,192.168.1.99,10m' >> /etc/dnsmasq.conf
To tell dnsmasq which ethernet interface it can and cannot listen on.
root@testing-harness:~# echo 'interface=eno1' >> /etc/dnsmasq.conf
During the creation of this document the interface eno1 is the unique one connected to AFT net.
===== AFT and its relation with dnsmasq.leases =====
As of dhcp server is up and running. AFT shall utilize dhcp-server's active ip leasing information to determine device IP address of PC-devices over AFT net. By default it is found upon /var/lib/misc/dnsmasq.leases
root@testing-harness:~# cat /var/lib/misc/dnsmasq.leases
1417132679 78:c5:e5:b4:4d:07 192.168.1.99 minox-fbe390b0bb404da *
1417134678 20:54:76:e4:d8:a5 192.168.1.57 minoy-ab6e5d0a524c9c7d *
......
.......
........ and so on...
Above a common dnsmasq.leases file.
=== Setup USB stick ===
==== Write support image on USB stick ====
A Debian operative system is used as support image for minnowboard. As of the memory stick is connected, it is required to verify how it was detected by OS. It is simple by using dmesg command.
root@testing-harness:~# dmesg
[354629.272050] usb 3-6: Product: ADATA USB Flash Drive
[354629.272051] usb 3-6: Manufacturer: ADATA
[354629.272052] usb 3-6: SerialNumber: 25C1008050080204
[354629.272594] usb-storage 3-6:1.0: USB Mass Storage device detected
[354629.272701] scsi host7: usb-storage 3-6:1.0
[354630.271607] scsi 7:0:0:0: Direct-Access ADATA USB Flash Drive 1.00 PQ: 0 ANSI: 6
[354630.271921] sd 7:0:0:0: Attached scsi generic sg1 type 0
[354630.272050] sd 7:0:0:0: [sdb] 30310400 512-byte logical blocks: (15.5 GB/14.5 GiB)
[354630.272189] sd 7:0:0:0: [sdb] Write Protect is off
[354630.272192] sd 7:0:0:0: [sdb] Mode Sense: 23 00 00 00
[354630.272329] sd 7:0:0:0: [sdb] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
[354630.278200] sdb: sdb1 sdb2 sdb3
[354630.279114] sd 7:0:0:0: [sdb] Attached SCSI removable disk
From this point onwards we are ready to record the image over USB stick 
root@testing-harness:~# wget -qO- $CI_SUPPORT_IMAGES_URL/minnowUSB.image.tar.gz | gunzip | tar -xOvf - | dd of=/dev/sdb
root@testing-harness:~# sync
Now Reboot the device and from BIOS menu select EFI USB Device from Boot Manager option.
==== Stuff available and steady configurations on support image ====
We should find these directories upon /mnt. The first one is utilized to mount the network share directory /home/tester of testing-harness
root@Debian-Minnowmax:~# ls /mnt
img_data_nfs super_target_root target_root
We should find firmware required for Realtek Ethernet card that is utilizing minnow-board max
root@Debian-Minnowmax:~# dpkg -l firmware-realtek*
The support image can be accessed over SSH without a password (it was achieved by injecting a known public SSH-key to /root/.ssh/authorized_keys). In other hand, config file contains info about the private key to be used when connecting with other hosts. There is also enabled the ssh access to this support image as root.
root@Debian-Minnowmax:~# ls .ssh
authorized_keys config id_rsa_testing_harness.pub id_rsa_testing_harness
root@Debian-Minnowmax:~# tail -n3 /etc/ssh/sshd_config
# ADDITIONS BY SUPPORT IMAGE INSTALLATION SCRIPT:
PermitRootLogin yes
UsePAM no
Additionally we should find the following debian packages installed.
root@Debian-Minnowmax:~# dpkg -l nfs-common nfs-server bmap-tools openssh-server vim python python-pip vim
bash-completion nano initramfs-tools net-tools ntp locales parted attr tree
==== Required configurations on support image ====
===== Logging on support image =====
As of support image has booted, it is posible to login as root by typing the password rootme
===== Mounting harness shared directory '/home/tester' =====
Edit /etc/fstab to mount remote directory tester at every time when booting operative system. 
192.168.1.1:/home/tester /mnt/img_data_nfs nfs rsize=8192,wsize=8192,timeo=14,intr,nolock,auto
===== Change network device name from eth1 back to eth0 =====
The interface name of a network device increases if the mac address of the physical network card changes. A common case is when booting a computer with an image created over a different one. It is to fix it by editing '/etc/udev/rules.d/70-persistent-net.rules', (Copy the new mac address to the line of your eth0 rule and delete the new rule for eth1).
# PCI device 0x15ad:0x07b0 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:13:20:fe:33:72",
 ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
=== Install and configure AFT ===
This section assumes that the testing harness and USB stick have already been setup.
==== Installation ====
AFT has to be installed upon testing-harness hosts by using the root user. AFT can be both installed and updated the same way: By cloning this repository or pulling in the latest changes and executing build and install subcommands with the setup.py installer
root@testing-harness:~# git clone https://github.com/01org/AFT
root@testing-harness:~# cd AFT && python setup.py build 
root@testing-harness:~# python setup.py install
==== Configuration ====
===== Edit platform.cfg =====
By default the file /etc/aft/devices/platform.cfg come with several sections predefined. Each section was created for a platform family.
Minnowboard belongs to PC platform family. The explanation of each field is the file comments.
[PC]
; PC Family are platforms which have a network interface with
; a fixed MAC-address and usually have a some sort of
; BIOS/EFI menu.
; Utilized by AFT to know ip address of any DUTs over AFT net
leases_file_name = /var/lib/misc/dnsmasq.leases
; keyboard sequence that will be used during configuration
; to check and verify that the keyboard emulator is functional
; (!!LATEST AFT COMMIT HAS REMOVED THIS BASIC HEALTH CHECK!!)
config_check_keystrokes = /home/tester/keystrokes/empty
[Edison]
[BeagleBoneBlack]
leases_file_name = /var/lib/misc/dnsmasq.leases
===== The keystroke files =====
Only two keystroke files are required.
# One to boot with the support image (AKA booting in 'service/support mode').
# Another one to boot with the target image (AKA booting in 'test mode')
During the creation of this document we have stored and created the keystroke files within '/home/tester/keystrokes' (By using tester user we have created the directory)
tester@testing-harness:~/keystrokes$ more service_mode.json 
{
 "ktasks": {
 "seq": [
 {
 "_desc": "EFI_Shell",
 "_inst": "<f2>",
 "_time_gap": "1",
 "_times": "5"
 },
 {
 "_desc": "Enter_EFI",
 "_inst": "<enter>",
 "_time_gap": "1",
 "_times": "1"
 },
 {
 "_desc": "Skip_startup",
 "_inst": "<esc>",
 "_time_gap": "1",
 "_times": "1"
 },
 {
 "_desc": "Select usb stick as boot device",
 "_inst": "fs1:",
 "_time_gap": "1",
 "_times": "1"
 },
 {
 "_desc": "Enter_device",
 "_inst": "<enter>",
 "_time_gap": "1",
 "_times": "1"
 },
 {
 "_desc": "EFI_Location",
 "_inst": "efi\\boot\\bootx64.efi",
 "_time_gap": "0",
 "_times": "1"
 },
 {
 "_desc": "Enter_File",
 "_inst": "<enter>",
 "_time_gap": "2",
 "_times": "1"
 },
 {
 "_desc": "Select first grub option",
 "_inst": "<enter>",
 "_time_gap": "10",
 "_times": "1"
 }
 ]
 }
}
tester@testing-harness:~/keystrokes$ more test_mode.json 
{
 "ktasks": {
 "seq": [
 {
 "_desc": "EFI_Shell",
 "_inst": "<f2>",
 "_time_gap": "1",
 "_times": "5"
 },
 {
 "_desc": "Enter_EFI",
 "_inst": "<enter>",
 "_time_gap": "1",
 "_times": "1"
 },
 {
 "_desc": "Skip_startup",
 "_inst": "<esc>",
 "_time_gap": "1",
 "_times": "1"
 },
 {
 "_desc": "Select SD CARD as boot device",
 "_inst": "fs0:",
 "_time_gap": "1",
 "_times": "1"
 },
 {
 "_desc": "Enter_device",
 "_inst": "<enter>",
 "_time_gap": "1",
 "_times": "1"
 },
 {
 "_desc": "EFI_Location",
 "_inst": "efi\\boot\\bootx64.efi",
 "_time_gap": "0",
 "_times": "1"
 },
 {
 "_desc": "Enter_File",
 "_inst": "<enter>",
 "_time_gap": "2",
 "_times": "1"
 },
 {
 "_desc": "Select first grub option",
 "_inst": "<enter>",
 "_time_gap": "10",
 "_times": "1"
 }
 ]
 }
}
===== Edit catalog.cfg =====
By default the file /etc/aft/devices/catalog.cfg come with several sections predefined. Each section represents a supported device by AFT. It is required to set the above keystroke files over minnow-board max section. It is also required to set the cutter_type to 'netbootercutter' (It is the cutter type which represents the remote power switch utilized over AFT net)
[MinnowboardMAX]
platform = PC
keyboard_emulator = km232keyboard
cutter_type = netbootercutter
test_plan = iot_qatest
target_device = /dev/mmcblk0
root_partition = /dev/mmcblk0p2
service_mode = Debian
test_mode = yocto
service_mode_keystrokes = /home/tester/keystrokes/service_mode.json
test_mode_keystrokes = /home/tester/keystrokes/test_mode.json
serial_bauds = 115200
===== Edit topology.cfg =====
Specific details, such as power cutter slot where minnow-board is connected and minnow-board max address should be set in 
/etc/aft/devices/topology.cfg
[MINNOWBOARD1]
model = MinnowboardMAX
id = 00:13:20:fe:33:72
netbooter_model = np0801dt
channel = 1 
The above model points to an entry in catalog.cfg.
== Runnig AFT with an Ostro image ==
Well this is the final step.
So it is required to count with an ostro image on bmap format
root@testing-harness:/home/tester# ls
ostro-image-swupd-dev-intel-corei7-64-2016-10-19_05-56-10-build-508.dsk
ostro-image-swupd-dev-intel-corei7-64-2016-10-19_05-56-10-build-508.dsk.bmap
To run AFT shall be utilized the next script
#!/bin/sh
# This image has to be hosted over tester user's home directory
TEST_IMAGE=ostro-image-swupd-dev-intel-corei7-64-2016-10-19_05-56-10-build-508.dsk
export http_proxy= && export HTTP_PROXY=
`which aft` minnowboardmax $TEST_IMAGE
By using sudo it'll be started
root@testing-harness:/home/tester# sudo ./test.sh
Finally the aft execution can be monitored by using tail command
root@testing-harness:/home/tester# tail -f aft.log
2016-10-24 16:26:51,482 - INFO - Built configuration sets for 5 devices
2016-10-24 16:26:51,488 - INFO - Attempting to acquire minnowboard1
2016-10-24 16:26:51,488 - INFO - Device acquired.
2016-10-24 16:26:51,489 - INFO - Built test plan with 1 test cases.
2016-10-24 16:26:51,489 - INFO - Trying to enter Debian mode up to 4 times.
2016-10-24 16:26:51,489 - INFO - Rebooting the device.
2016-10-24 16:27:05,575 - INFO - Using KM232Keyboard to send keyboard sequence /home/tester/keystrokes/service_mode.json
2016-10-24 16:27:38,095 - INFO - Waiting for the device to become responsive
2016-10-24 16:27:48,114 - WARNING - Could not establish ssh-connection to 192.168.1.79. SSH return code: 255.
2016-10-24 16:28:01,201 - WARNING - Could not establish ssh-connection to 192.168.1.79. SSH return code: 255.
2016-10-24 16:28:14,288 - WARNING - Could not establish ssh-connection to 192.168.1.79. SSH return code: 255.
2016-10-24 16:28:24,736 - INFO - Got a response from 192.168.1.79
2016-10-24 16:28:25,090 - INFO - Found device in Debian mode.
2016-10-24 16:28:25,091 - INFO - Mounting the nfs containing the image to flash.
2016-10-24 16:28:25,531 - INFO - Writing /mnt/img_data_nfs/ostro-image-swupd-dev-intel-corei7-64-2016-10-19_05-56-10-build-508.dsk to internal storage.
2016-10-24 16:28:25,531 - INFO - Found ostro-image-swupd-dev-intel-corei7-64-2016-10-19_05-56-10-build-508.dsk.bmap. Using bmap for flashing.
2016-10-24 16:32:26,794 - INFO - Partprobing.
2016-10-24 16:32:29,344 - INFO - Mount one layer.
2016-10-24 16:32:30,110 - INFO - Writing ssh-key to device.
2016-10-24 16:32:31,474 - INFO - Adding IMA attribute to the ssh-key
2016-10-24 16:32:31,844 - INFO - Flushing.
2016-10-24 16:32:32,264 - INFO - Unmounting.
2016-10-24 16:32:32,644 - INFO - Executing the test plan
2016-10-24 16:32:32,644 - INFO - Test plan start time: 1477344752.6447997
2016-10-24 16:32:32,645 - INFO - Executing test case 1 of [<aft.testcases.qatestcase.QATestCase object at 0x7f14d26ad588>]
2016-10-24 16:32:32,645 - INFO - Test case start time: 2016-10-24 16:32:32.645253
2016-10-24 16:32:32,645 - INFO - Trying to enter yocto mode up to 4 times.
2016-10-24 16:32:32,645 - INFO - Rebooting the device.
2016-10-24 16:32:46,694 - INFO - Using KM232Keyboard to send keyboard sequence /home/tester/keystrokes/test_mode.json
2016-10-24 16:33:19,183 - INFO - Waiting for the device to become responsive
2016-10-24 16:33:29,207 - WARNING - Could not establish ssh-connection to 192.168.1.79. SSH return code: 255.
2016-10-24 16:33:40,689 - WARNING - Could not establish ssh-connection to 192.168.1.79. SSH return code: 255.
2016-10-24 16:33:52,977 - WARNING - Could not establish ssh-connection to 192.168.1.79. SSH return code: 255.
2016-10-24 16:34:05,920 - INFO - Got a response from 192.168.1.79
2016-10-24 16:34:06,125 - INFO - Found device in yocto mode.
2016-10-24 16:34:06,642 - INFO - python: can't open file 'iottest/runtest.py': [Errno 2] No such file or directory
2016-10-24 16:34:06,642 - INFO - Test Duration: 0:01:33.997627
2016-10-24 16:34:06,643 - INFO - Test plan end time: 1477344846.6430657
2016-10-24 16:34:06,643 - INFO - Storing the test results.
2016-10-24 16:34:06,643 - INFO - Results saved to /home/tester/results.xml.