Repository: siderolabs/talos

Issue #8367: Volume Management
Created: 2024-02-26T11:15:41Z
Closely related: #8016
Problem Statement
Talos Linux is not flexible in the way it manages volumes, it occupies the whole system disk, creating an EPHEMERAL partition covering 99% of the disk space. User disk management is fragile, requires extra steps to get it to work properly (mounting into the kubelet), doesn’t support wiping disks, etc. Talos does not detect properly various partition types which leads to wiping user data (e.g. Ceph bluestore).
There were following requests from the users/customers which can’t be addressed in the current design:

running Talos network booted (i.e. with STATE / EPHEMERAL on tmpfs)
running Talos installed e.g. to the SBC’s SD card, but mounting /var from an NVMe/SSD
Azure VMs with directly attached NVMes will always install Talos to the network volume, so locally attached NVMe can’t be used for e.g. containerd state
splitting etcd data directory to a separate disk
performing wipe operations on the contents of the /var:

wiping system directories (e.g. containerd state)
wiping user data (e.g. /var/mnt/foo)


disk encryption of the user disks (volumes)
read-only user disks mounts (e.g. mounting my precious photo archive to the Talos machine, making sure that Talos never touches the contents)
volume/disk management operations without reboot:

wiping user disks/volumes
wiping system areas


creating additional user-managed partitions on the system disk:

/data
swap space


container image cache storage
some persistent across reboots logs storage (e.g. storing installation logs during staged upgrades)

The proposed design provides an option to solve the issues mentioned above.
Groundwork
Before we move into volume management operations, there is some amount of work that needs to be done to improve the blockdevice management operations:

Talos should quickly and easily detect various filesystem/partition types, including the most common ones, the ones that can be used in Kubernetes clusters. The minimum detection of the filesystem/partition prevents the disk from being considered empty and eligible for allocation
The blockdevices/partitions should be presented as resources in a way that it allows to render them as a tree presenting a user with a view of available block devices, their types, partitions, filesystem types, available space, etc.
Talos should detect and reliably show information which matches various standard Linux tools, e.g. blkid, to allow easier identification of storage objects.

Installation Process
Talos installation should do the bare minimum to make sure that Talos can be booted from the disk, without touching the pieces which are not strictly required to boot Talos. This might include installing Talos without having machine configuration.
So the install should only touch following partitions/objects:

BOOT / EFI partitions (boot assets, boot loader itself)
META partition
bootloader-specific stuff, e.g. writing to the MBR

Any management of the storage/volumes should be deferred to the Talos running on the host (i.e. creating /var, /system/state, etc.)
Volumes
Let’s introduce a new concept of the volumes, which will solve the ideas mentioned above and allow us to take storage management to the next level.
There are two kinds of volumes:

system volumes (i.e. required by Talos, and Talos provides default configuration for them if no other configuration is available)
user volumes (configured and managed by users, optional)

Every volume has several most important features:

lookup: Talos can find the volume by some selector, or say that the volume is not available
provisioning (optional): if the volume is not available, Talos can provision the volume (e.g. create a partition), so that it can be looked up, and it becomes available
parent: if the volume has a parent volume, it creates a dependency relationship between them
mount path: might create another dependency on the volume which provides the mount path

Volumes support a basic set of operations:

Mount/Unmount
Wipe (requires unmounting first)
Destroy (implies Wipe, but removes provisioned volumes)

Volume types:

disk (e.g. use the whole disk)
partition (allocate a partition on the disk)
subdirectory (a sub-path on other Volume)

Volume formats:

filesystem (or none)
encryption (or none)
…

Volume additional options:

criticality (should be available for the pods to be started)
mounted into the kubelet

System Volumes
As of today, Talos implicitly has the following volume types:



Name
Lookup
Provisioning
Format




STATE
a partition with the label STATE
create a partition on the system disk of size X MiB
xfs, optionally encrypted


EPHEMERAL
a partition with the label EPHEMERAL
create a partition on the system disk which occupies all remaining space
xfs, optionally encrypted


etcd data
-
subdirectory of EPHEMERAL, /var/lib/etcd
-


containerd data
-
subdirectory of EPHEMERAL, /var/lib/containerd
-


kubelet data
-
subdirectory of EPHEMERAL, /var/lib/kubelet
-



Volume Lifecycle
Talos services can express their dependency on the volumes. For example, kubelet service can only be started when kubelet data volume is available. Same way, if kubelet data volume is going to be unmounted, kubelet should be stopped first.
The boot process should naturally stop when the required volume is not available. E.g. maintenance mode of Talos implies that the boot can’t proceed as long as the volume configuration is not available.
Volume Configuration
System volumes have implicit configuration, which is applied as long as v1alpha1.Config is applied to the machine. Some properties are configurable in v1alpha1.Config,  e.g. disk encryption. If an explicit volume configuration is provided, Talos uses that.
For example, if the user configures EPHEMERAL to be tmpfs of size 10 GiB, it will be created on each boot as instructed.
Users might provide configuration for user volumes (similar to the user disks feature today), which might be critical for the pods to be started, an otherwise e.g. extension services might provide a dependency on the additional volumes.
Some system volumes might be optional, i.e. configured by the users - for example, container image cache.
Upgrades and Wiping
Talos Linux upgrades should not wipe anything by default, and wiping should be an additional operation which can be done without an upgrade, or can optionally be combined with an upgrade.
Update itself should only modify boot assets/boot loader, i.e. ensure that the new version of Talos Linux can be booted up from the disk device.
Wiping is volume-based, examples:

I want to wipe EPHEMERAL, which implies wiping all volumes which have EPHEMERAL as a parent (e.g. subdirectory volume of /var/lib/etcd); all services which depend on EPHEMERAL or its children should be stopped, but reboot is not necessary, as the EPHEMERAL will be re-provisioned after the wipe
I want to wipe etcd data, which in the default configuration implies leaving etcd, stopping etcd services, performing rm -rf /var/lib/etcd, and re-starting etcd join process

Notes
As pointed out by @utkuozdemir, EPHEMERAL might be a bad name given that the partition is not supposed to be forced wiped by default.

  
    
  
      
        
    

      
    
      The content you are editing has changed. Please copy your edits and refresh the page.

    
      
  
    
    

  
        
          
              
      
        Title of tasklist
    
  
    
      

      
  
    

  
    

  

    


              
    Cancel
  


              
    Save
  


    
      Tasks
            
    

Edit tasklist title




      
        
  Preview

          Give feedback

            
  
      
    

Tasklist Tasks, more options



  
    
                
  
    
        
    
    
    
      
        
                          
    

                Rename

      

  

        
    
    
    
      
        
                          
    

                
                  Copy markdown
                

      

  

        
    
    
    
      
        
                          
    

                Delete tasklist

      

  

    




      
  

            

  
    
  
    
      
        Delete tasklist
      
        Delete tasklist block?
    
    
      
    

    
  
  

      
        Are you sure? All relationships in this tasklist will be removed.
      
                        
    Cancel
  


                  
    Delete
  




    

    
          
          
            
    

          
      
            
          
    


    
        v2: Talos Volume support

      go-blockdevice#78

    
  

      
        
          
    
  
      

  
      
    
    
  


      7 of 8
  




        
      

    
      
      
    

    
      

        

            +0
      
        
        
  
    
                

              


      

      
        
  
      
      
        
      


      
        
  
      
    

Edit...



  
    
                
  
    
        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      


          
          
            
    

          
      
            
          
    


    
        implement configuration for EPHEMERAL volume

      #9261

    
  

      
        
          
    


        
      

    
      
      
    

    
      

        

            +0
      
        
        
  
    
                

              


      

      
        
  
      
      


      
        
  
      
    

Edit...



  
    
                
  
    
        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      


          
          
            
    

          
      
            
          
    


    
        remove v1 pkg/mount

      #9471

    
  

      
        
          
    


        
      

    
      
      
    

    
      

        

            +0
      
        
        
  
    
                

              


      

      
        
  
      
      


      
        
  
      
    

Edit...



  
    
                
  
    
        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      


          
          
            
    

          
          
            
              
              remove v1 go-blockdevice
            

            
              
                remove v1 go-blockdevice

              
                remove v1 go-blockdevice
            
          



      
        
  
      
    

Options



  
    
                
  
    
        
    
    
    
      
        
                        
                  Convert to issue

      

  

        
    
    
    
      
        
                        
                  Toggle completion

      

  

        
    
    
    
      
        
                        
                  Rename

      

  

        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      


          
          
            
    

          
      
            
          
    


    
        implement mount controller

      #9602

    
  

      
        
          
    


        
      

    
      
      
    

    
      

        

            +0
      
        
        
  
    
                

              


      

      
        
  
      
      


      
        
  
      
    

Edit...



  
    
                
  
    
        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      


          
          
            
    

          
          
            
              
              implement volume wipe
            

            
              
                implement volume wipe

              
                implement volume wipe
            
          



      
        
  
      
    

Options



  
    
                
  
    
        
    
    
    
      
        
                        
                  Convert to issue

      

  

        
    
    
    
      
        
                        
                  Toggle completion

      

  

        
    
    
    
      
        
                        
                  Rename

      

  

        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      


          
          
            
    

          
      
            
          
    


    
        support wiping non-volume blockdevices

      #9731

    
  

      
        
          
    


        
      

    
      
      
    

    
      

        

            +0
      
        
        
  
    
                

              


      

      
        
  
      
      


      
        
  
      
    

Edit...



  
    
                
  
    
        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      


          
          
            
    

          
      
            
          
    


    
        implement wipe by probe signatures

      #9724

    
  

      
        
          
    


        
      

    
      
      
    

    
      

        

            +0
      
        
        
  
    
                

              


      

      
        
  
      
      


      
        
  
      
    

Edit...



  
    
                
  
    
        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      


          
          
            
    

          
          
            
              
              implement directory volumes
            

            
              
                implement directory volumes

              
                implement directory volumes
            
          



      
        
  
      
    

Options



  
    
                
  
    
        
    
    
    
      
        
                        
                  Convert to issue

      

  

        
    
    
    
      
        
                        
                  Toggle completion

      

  

        
    
    
    
      
        
                        
                  Rename

      

  

        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      


          
          
            
    

          
          
            
              
              user volumes
            

            
              
                user volumes

              
                user volumes
            
          



      
        
  
      
    

Options



  
    
                
  
    
        
    
    
    
      
        
                        
                  Convert to issue

      

  

        
    
    
    
      
        
                        
                  Toggle completion

      

  

        
    
    
    
      
        
                        
                  Rename

      

  

        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      


          
          
            
    

          
      
            
          
    


    
        "Error probing disks" after upgrade from 1.7.6 to 1.8.1

      #9530

    
  

      
        
          
    


        
      

    
      
      
    

    
      

        

            +0
      
        
        
  
    
                

              


      

      
        
  
      
          
    



      
        
  
      
    

Edit...



  
    
                
  
    
        
    
    
    
      
        
                      
                  Remove

      

  

    




      
  
      
      
        
          
    

        
        Successfully updated the issue's project
      
      
        
          
    

        
        There was an error updating the issue's project
      



        
          
            
    

          
          
            
              
              
            

            
              
                

              
                
            
          
          
            
  
      
    

Options



  
    
                
  
    
        
    
    
    
      
        
                          
                    Convert to issue

      

  

        
    
    
    
      
        
                        
                  Toggle completion

      

  

        
    
    
    
      
        
                        
                  Rename

      

  

        
    
    
    
      
        
                        
                    Remove

      

  

    




      
  
          
        
        
          
            
    

          
          
            
          
          
        
    

          
    
      
        
    

      
    Add item to  Tasks
  


  
    
      
        
          
  
    
      Type to add an item or paste in an issue URL
    
      
        
          
    

        
        
      
  
  
  
        
      
  
  
  
    
    
    Loading

Duplicate issue #6325: Feature request: better disk management
Created: 2022-09-25T14:46:46Z
Feature Request
Description
Talos has config for specifying attached disks and their partitions, and will automatically partition and format them when it boots with an uninitialized disk present. This makes working with Ceph tricky, since Ceph wants an uninitialized disk (or partition) to itself.
I recently managed to set up Ceph in a trial home environment on an empty partition (sliced off from a local disk, of which I gave 50GB over to containerd), and it works fine -- but the process of setting that up required manual steps in privileged containers.
It would be nice if Talos provided some extra functionality here:

The ability to specify a filesystem to format a partition with in the machine config (with "none" as an option)
Possibly CLI commands to deal with creating, deleting, and formatting local disks. It clearly already knows how to do these things, so it would be helpful to do them on request, instead of just on boot.

Closing comment (2024-07-09T10:28:56Z): Dup of #8367