Repository: archlinux/archinstall

Issue:
Parted doesn't work with loop devices
This isn't an issue for the installer per se but rather for testing. 
The README instructions currently refer to creating an image and then setting it up as a loop device. But such device can't be listed by parted and therefore also not pyparted and archinstall.

https://git.savannah.gnu.org/gitweb/?p=parted.git;a=blob;f=libparted/arch/linux.c;h=ccbba8656bc7ef0663aea6431bfb8873deccd0ef;hb=HEAD#l2304

Maybe a question for @Torxed would you be aware of other options to get a test device setup that could be used in tests?


Discussion:
I'll have a dig at it when I come back home this weekend :) 
I just finished typing this up, took some time to put it all down. Let me know if it was worth it.

What @svartkanin linked is just a function to skip listing them. Partitioning loop devices with parted is possible. Here is an example that includes listing a loop device with `lsblk` and `losetup` as well as partitioning it with `parted`.

```
# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    254:0    0   20G  0 disk
├─vda1 254:1    0    1G  0 part
└─vda2 254:2    0   19G  0 part /
# losetup -a
# truncate -s 2G example.img
# losetup -P --show -f example.img
/dev/loop0
# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0    7:0    0    2G  0 loop
vda    254:0    0   20G  0 disk
├─vda1 254:1    0    1G  0 part
└─vda2 254:2    0   19G  0 part /
# losetup -a
/dev/loop0: [65026]:393281 (/root/example.img)
# parted --script /dev/loop0 \
> mklabel gpt \
> unit mib \
> mkpart boot fat32 1 513 \
> mkpart root ext4 513 100% \
> set 1 boot on
# lsblk
NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0       7:0    0    2G  0 loop
├─loop0p1 259:0    0  512M  0 part
└─loop0p2 259:1    0  1.5G  0 part
vda       254:0    0   20G  0 disk
├─vda1    254:1    0    1G  0 part
└─vda2    254:2    0   19G  0 part /
# losetup -a
/dev/loop0: [65026]:393281 (/root/example.img)
```

It has been a while but I had this working in my testing. Here is the first crude patch I used to add all loop devices to the list of available devices: https://github.com/archlinux/archinstall/commit/51779582675ba2740f848c1414c718f20ad9073f. There is more testing I did locally and in a private repo to work on the details. I am still interested in getting this together and was waiting for pull request merges to pick back up before adding pull requests.

What I have discovered is it may not be possible to do within a Docker image like is being used for the other GitHub workflows for the repository. The gist is there does not appear to be the ability to get updated information about the loop device partitions after creating them (no UUID or PARTUUID available I think maybe other stuff or everything, I do not recall the exact details).

The workaround I used was to [create a chroot](https://wiki.archlinux.org/title/Install_Arch_Linux_from_existing_Linux#Creating_a_chroot) of Arch Linux in a GitHub Ubuntu virtual machine (not a Docker image) workflow. I used _Method B_ since archinstall will generally be run from the LiveCD image and it will already have all of the needed packages.

So with a patch to add loop devices to the list of all devices, and scripting to set up the chroot and prepare the image file, it is possible within a GitHub workflow. From there you can produce an image file of the installation that can be booted with QEMU in a GitHub Ubuntu vm workflow or the same workflow the image was created in. As @Torxed pointed out here #1114, the Arch Linux ci script [build_in_archiso_vm.sh](https://gitlab.archlinux.org/archlinux/ci-scripts/-/blob/master/scripts/build_in_archiso_vm.sh) is a good way to do this. I used that with some inconsequential modifications in my testing.

Also to get the resulting image to boot with QEMU I had to boot a fallback initramfs (it has "a full range of modules") or configure mkinitcpio to have the additional hook `ata_piix` and generate the initramfs again. If not the boot loader would not detect the drive. Might just be how QEMU was configured for the drive in my command line arguments?

Maybe Arch Linux DevOps can set up an Arch Linux virtual machine as a runner?
I'll for sure ask them, I know this would be very possible on the GitLab instance and other things are prepped for a move there at a later time (currently very early discussions because the community here is very awesome.. so no move yet).

They have been asked now :) great work and I'd love a runner with your work in it! The quemu arguments probably matter, but mostly the PCI bus and blockdev setup if everything is set up as 1:1 to match the host in the other regards :)
@codefiles I probably didn't describe it properly but yes what I meant is that `-l` wouldn't list the partitions with `parted` and therefore archinstall wouldn't be able to list them as options. 
But your change seems to address that, even tho I was hoping to be able to avoid such "manually" detection using a library now :( but there doesn't seem to be a way around it 
> The quemu arguments probably matter, but mostly the PCI bus and blockdev setup if everything is set up as 1:1 to match the host in the other regards :)

@Torxed I may be mistaking testing I did locally with QEMU where I needed to boot a fallback initramfs with testing I also did running it remote as a workflow. I probably created an install using a physical machine as a host with the image to be booted as a VM in QEMU. In that case mkinitcpio would pick up the host machine modules instead of modules it would pick up, and possible need, if it was running in QEMU. That was probably the issue I am thinking about with the `ata_piix` module.

> @codefiles I probably didn't describe it properly but yes what I meant is that -l wouldn't list the partitions with parted and therefore archinstall wouldn't be able to list them as options.

@svartkanin No you explained it fine. I figured you might have thought that they would need to be in the listing to partition with parted/pyparted.

>But your change seems to address that, even tho I was hoping to be able to avoid such "manually" detection using a library now :( but there doesn't seem to be a way around it

What library? It works with `losetup` which is in the util-linux package. That package is already required and is in the PKGBUILD dependency list. Could also work with `lsblk` which is in use in the code and is from util-linux too.
I meant to use the pyparted library alone and that the getAllDevices will just magically find everything so we don't have to SysCommand things :)
@codefiles do you want to raise a PR to get the loop device handler in (the commit you linked)? Maybe with a try-except and only logging any problems but just continuing with the other devices?
Yea, I will submit it in a bit. Good idea on the try-except and logging, thanks for the suggestion.
> What I have discovered is it may not be possible to do within a Docker image like is being used for the other GitHub workflows for the repository. The gist is there does not appear to be the ability to get updated information about the loop device partitions after creating them (no UUID or PARTUUID available I think maybe other stuff or everything, I do not recall the exact details).

Assuming you are hitting this issue https://github.com/moby/moby/issues/27886, try mounting `/dev:/dev` in the container as a workaround (syntax documented [here](https://docs.github.com/en/actions/using-jobs/running-jobs-in-a-container#mounting-volumes-in-a-container)).
I am aware of that and tried it, https://github.com/archlinux/archinstall/commit/64e7865e2cf8d075ade23da9abdb3ebc00bbf97b. Using that suggestion, it is then possible to see the loop devices but as I stated it is not possible to get new information about partitions that were created. Could you create a simple test example to demonstrate use of the information you provided? In an Arch Linux Docker container create an image preferably with `truncate`, then set up the image as a loop device with `losetup`, partition it and format the partition(s). While still in the Docker container print the UUID and PARTUUID for the new partition(s). Finally can you run the `mount` command (as archinstall will attempt to) inside the Docker container and mount the loop device partition(s)?

Resolving patch:
diff --git a/archinstall/lib/disk/device_handler.py b/archinstall/lib/disk/device_handler.py
--- a/archinstall/lib/disk/device_handler.py
+++ b/archinstall/lib/disk/device_handler.py
@@ -10,7 +10,7 @@
 from parted import (  # type: ignore
 	Disk, Geometry, FileSystem,
 	PartitionException, DiskLabelException,
-	getAllDevices, freshDisk, Partition, Device
+	getDevice, getAllDevices, freshDisk, Partition, Device
 )
 
 from .device_model import (
@@ -45,7 +45,18 @@ def devices(self) -> List[BDevice]:
 	def load_devices(self):
 		block_devices = {}
 
-		for device in getAllDevices():
+		devices = getAllDevices()
+
+		try:
+			loop_devices = SysCommand(['losetup', '-a'])
+		except SysCallError as err:
+			debug(f'Failed to get loop devices: {err}')
+		else:
+			for ld_info in str(loop_devices).splitlines():
+				loop_device = getDevice(ld_info.split(':', maxsplit=1)[0])
+				devices.append(loop_device)
+
+		for device in devices:
 			if get_lsblk_info(device.path).type == 'rom':
 				continue
 
diff --git a/archinstall/lib/disk/device_model.py b/archinstall/lib/disk/device_model.py
--- a/archinstall/lib/disk/device_model.py
+++ b/archinstall/lib/disk/device_model.py
@@ -374,7 +374,10 @@ def table_data(self) -> Dict[str, Any]:
 	@classmethod
 	def from_disk(cls, disk: Disk) -> _DeviceInfo:
 		device = disk.device
-		device_type = parted.devices[device.type]
+		if device.type == 18:
+			device_type = 'loop'
+		else:
+			device_type = parted.devices[device.type]
 
 		sector_size = Size(device.sectorSize, Unit.B)
 		free_space = [DeviceGeometry(g, sector_size) for g in disk.getFreeSpaceRegions()]


Test patch: