Repository: dell/csi-baremetal
Pull request #490 (2021-08-19 08:16:26)

[ISSUE-489] re-activate volume group on disk reinsert
## Purpose
### Issue #489 

Re-activate volume group on drive transition from OFFLINE to ONLINE

## PR checklist
- [x] Add link to the issue
- [x] Choose PR label
- [x] New unit tests added
- [x] Modified code has meaningful comments
- [ ] All TODOs are linked with the issues
- [ ] All comments are resolved

## Testing
- Simulate disk re-insert:
```
borism1@ubuntu:/workspace/csi-baremetal$ drives | grep VDH1H03D
ecdcc891-62d4-48b6-bfc1-afedba53f36b   8001000000000   HDD    VDH1H03D           UNKNOWN   IN_USE   OFFLINE   <none>   ee54707c-2252-4a6a-8bc6-40951c741615
borism1@ubuntu:/workspace/csi-baremetal$ drives | grep VDH1H03D
ecdcc891-62d4-48b6-bfc1-afedba53f36b   8001000000000   HDD    VDH1H03D           GOOD     IN_USE   ONLINE   <none>   ee54707c-2252-4a6a-8bc6-40951c741615
```
- Check for LVG events:
```
borism1@ubuntu:/workspace/csi-baremetal$ kubectl describe lvg 3b7d6b51-12f7-48a8-a438-1c6e3fa1ff69
Name:         3b7d6b51-12f7-48a8-a438-1c6e3fa1ff69
...
Events:
  Type     Reason                         Age   From                                                      Message
  ----     ------                         ----  ----                                                      -------
  Normal   VolumeGroupScanInvolved        7s    csi-baremetal-node, ee54707c-2252-4a6a-8bc6-40951c741615  Check for IO errors
  Warning  VolumeGroupReactivateInvolved  7s    csi-baremetal-node, ee54707c-2252-4a6a-8bc6-40951c741615  IO errors detected
```
- Restart pod to make sure that mount will pass:
```
borism1@ubuntu:/workspace/csi-baremetal$ kubectl delete pod web-3
pod "web-3" deleted
borism1@ubuntu:/workspace/csi-baremetal$ volumes | grep 3b7d6b51-12f7-48a8-a438-1c6e3fa1ff69
default     pvc-c1919fc9-304e-4462-92ce-841f2986c6c6   1002438656   xfs    GOOD     OPERATIVE   PUBLISHED   IN_USE   3b7d6b51-12f7-48a8-a438-1c6e3fa1ff69   ee54707c-2252-4a6a-8bc6-40951c741615
default     pvc-c665c4f6-0881-454e-b531-10883f31aa55   1002438656   xfs    GOOD     OPERATIVE   PUBLISHED   IN_USE   3b7d6b51-12f7-48a8-a438-1c6e3fa1ff69   ee54707c-2252-4a6a-8bc6-40951c741615
default     pvc-7213ac7f-c7e3-4297-ac8f-30fd6f1e93b7   1002438656   xfs    GOOD     OPERATIVE   PUBLISHED   IN_USE   3b7d6b51-12f7-48a8-a438-1c6e3fa1ff69   ee54707c-2252-4a6a-8bc6-40951c741615
default     pvc-b925398a-96e2-4119-bdff-b90ac5ee20ca   1002438656   xfs    GOOD     OPERATIVE   PUBLISHED   IN_USE   3b7d6b51-12f7-48a8-a438-1c6e3fa1ff69   ee54707c-2252-4a6a-8bc6-40951c741615
default     pvc-2c6bdc9b-7519-4b7f-9ef9-130c6c3b6144   1002438656   xfs    GOOD     OPERATIVE   PUBLISHED   IN_USE   3b7d6b51-12f7-48a8-a438-1c6e3fa1ff69   ee54707c-2252-4a6a-8bc6-40951c741615
borism1@ubuntu:/workspace/csi-baremetal$ pods | grep web-3
web-3                                            1/1     Running   0          21s    192.168.195.172   provo-goop.ecs.lab.emc.com   <none>           <none>
```

codecov[bot]: # [Codecov](https://codecov.io/gh/dell/csi-baremetal/pull/490?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell) Report
> Merging [#490](https://codecov.io/gh/dell/csi-baremetal/pull/490?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell) (b3b1130) into [master](https://codecov.io/gh/dell/csi-baremetal/commit/a2a0c53e891cee42cfbd3980c067c41c063de639?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell) (a2a0c53) will **decrease** coverage by `0.13%`.
> The diff coverage is `40.74%`.

> :exclamation: Current head b3b1130 differs from pull request most recent head 6daee60. Consider uploading reports for the commit 6daee60 to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/dell/csi-baremetal/pull/490/graphs/tree.svg?width=650&height=150&src=pr&token=4D8WVNWX5O&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell)](https://codecov.io/gh/dell/csi-baremetal/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell)

```diff
@@            Coverage Diff             @@
##           master     #490      +/-   ##
==========================================
- Coverage   59.14%   59.01%   -0.14%     
==========================================
  Files          60       60              
  Lines        6002     6019      +17     
==========================================
+ Hits         3550     3552       +2     
- Misses       2246     2260      +14     
- Partials      206      207       +1     
```

| Flag | Coverage Δ | |
|---|---|---|
| unittests | `59.01% <40.74%> (-0.14%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/dell/csi-baremetal/pull/490?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell) | Coverage Δ | |
|---|---|---|
| [pkg/base/linuxutils/lvm/lvm.go](https://codecov.io/gh/dell/csi-baremetal/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell#diff-cGtnL2Jhc2UvbGludXh1dGlscy9sdm0vbHZtLmdv) | `85.50% <0.00%> (-8.15%)` | :arrow_down: |
| [pkg/node/volumemgr.go](https://codecov.io/gh/dell/csi-baremetal/pull/490/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell#diff-cGtnL25vZGUvdm9sdW1lbWdyLmdv) | `69.89% <73.33%> (-0.23%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/dell/csi-baremetal/pull/490?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/dell/csi-baremetal/pull/490?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell). Last update [a2a0c53...6daee60](https://codecov.io/gh/dell/csi-baremetal/pull/490?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dell).


Patch:
diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml
index b263e8b..5445b17 100644
--- a/.github/workflows/pr.yml
+++ b/.github/workflows/pr.yml
@@ -67,6 +67,7 @@ jobs:
 
     - name: Upload coverage report to codecov
       uses: codecov/codecov-action@v1.0.2
+      continue-on-error: true  # added here as workaround for codecov-action on validation of PR from forked repo
       with:
         token: ${{ secrets.CODECOV_TOKEN }}
         file: ./coverage.out
diff --git a/charts/csi-baremetal-driver/Chart.yaml b/charts/csi-baremetal-driver/Chart.yaml
index 4a466e3..4e57c79 100644
--- a/charts/csi-baremetal-driver/Chart.yaml
+++ b/charts/csi-baremetal-driver/Chart.yaml
@@ -1,5 +1,5 @@
 apiVersion: v2
-appVersion: 0.3.3
+appVersion: 0.4.0
 description: Bare-metal CSI driver
 name: csi-baremetal-driver
-version: 0.3.0
+version: 0.4.0
diff --git a/charts/csi-baremetal-operator/Chart.yaml b/charts/csi-baremetal-operator/Chart.yaml
index 937846e..bfc9274 100644
--- a/charts/csi-baremetal-operator/Chart.yaml
+++ b/charts/csi-baremetal-operator/Chart.yaml
@@ -1,5 +1,5 @@
 apiVersion: v2
-appVersion: 0.3.0
+appVersion: 0.4.0
 description: Bare-metal CSI Operator
 name: csi-baremetal-operator
-version: 0.3.0
+version: 0.4.0
diff --git a/charts/csi-baremetal-scheduler-extender/Chart.yaml b/charts/csi-baremetal-scheduler-extender/Chart.yaml
index 70204b2..009dc5f 100644
--- a/charts/csi-baremetal-scheduler-extender/Chart.yaml
+++ b/charts/csi-baremetal-scheduler-extender/Chart.yaml
@@ -1,5 +1,5 @@
 apiVersion: v2
-appVersion: 0.3.0
+appVersion: 0.4.0
 description: Bare-metal CSI Kubernetes Scheduler Extender
 name: csi-baremetal-scheduler-extender
-version: 0.3.0
+version: 0.4.0
diff --git a/charts/csi-baremetal-scheduler/Chart.yaml b/charts/csi-baremetal-scheduler/Chart.yaml
index 46bbc00..47dd084 100644
--- a/charts/csi-baremetal-scheduler/Chart.yaml
+++ b/charts/csi-baremetal-scheduler/Chart.yaml
@@ -1,5 +1,5 @@
 apiVersion: v2
-appVersion: 0.3.0
+appVersion: 0.4.0
 description: Bare-metal CSI Kubernetes Scheduler
 name: csi-baremetal-scheduler
-version: 0.3.0
+version: 0.4.0
diff --git a/pkg/base/constants.go b/pkg/base/constants.go
index ffe2d1e..5ef99b0 100644
--- a/pkg/base/constants.go
+++ b/pkg/base/constants.go
@@ -29,7 +29,7 @@ const (
 	PluginName = "csi-baremetal"
 	// PluginVersion is a version of current CSI plugin
 	// TODO: get rid of hardcoded value https://github.com/dell/csi-baremetal/issues/79
-	PluginVersion = "0.3.0"
+	PluginVersion = "0.4.0"
 	// DefaultDriveMgrEndpoint is the default gRPC endpoint for drivemgr
 	DefaultDriveMgrEndpoint = "tcp://:8888"
 	// DefaultHealthIP is the default gRPC IP for Health server
diff --git a/pkg/base/linuxutils/lvm/lvm.go b/pkg/base/linuxutils/lvm/lvm.go
index 9fcc1f8..677eb0b 100644
--- a/pkg/base/linuxutils/lvm/lvm.go
+++ b/pkg/base/linuxutils/lvm/lvm.go
@@ -21,6 +21,7 @@ package lvm
 import (
 	"errors"
 	"fmt"
+	"regexp"
 	"strconv"
 	"strings"
 	"time"
@@ -28,6 +29,7 @@ import (
 	"github.com/sirupsen/logrus"
 
 	"github.com/dell/csi-baremetal/pkg/base/command"
+	errTypes "github.com/dell/csi-baremetal/pkg/base/error"
 	"github.com/dell/csi-baremetal/pkg/base/util"
 )
 
@@ -46,6 +48,10 @@ const (
 	PVsListCmdTmpl = lvmPath + "pvdisplay --short"
 	// VGCreateCmdTmpl create VG on provided PVs cmd
 	VGCreateCmdTmpl = lvmPath + "vgcreate --yes %s %s" // add VG name and PV names
+	// VGScanCmdTmpl searches for all VGs
+	VGScanCmdTmpl = lvmPath + "vgscan"
+	// VGRefreshCmdTmpl reactivates an LV using the latest metadata
+	VGRefreshCmdTmpl = lvmPath + "vgchange --refresh %s"
 	// VGRemoveCmdTmpl remove VG cmd
 	VGRemoveCmdTmpl = lvmPath + "vgremove --yes %s" // add VG name
 	// AllPVsCmd returns all physical volumes on the system
@@ -71,6 +77,8 @@ type WrapLVM interface {
 	PVCreate(dev string) error
 	PVRemove(name string) error
 	VGCreate(name string, pvs ...string) error
+	VGScan(name string) (bool, error)
+	VGReactivate(name string) error
 	VGRemove(name string) error
 	LVCreate(name, size, vgName string) error
 	LVRemove(fullLVName string) error
@@ -150,6 +158,55 @@ func (l *LVM) VGCreate(name string, pvs ...string) error {
 	return err
 }
 
+// VGScan scans for all VGs and checks for IO errors for specific volume group name
+// Receives name of VG name to scan and check
+// Return boolean (false if no IO errors detected, true otherwise) and error if command failed to execute
+func (l *LVM) VGScan(name string) (bool, error) {
+	// scan and check for VG errors
+	var (
+		stdout  string
+		stderr  string
+		err     error
+		exp     *regexp.Regexp
+		ioError = "input/output error"
+	)
+	// do the scan
+	if stdout, stderr, err = l.e.RunCmd(VGScanCmdTmpl, command.UseMetrics(true),
+		command.CmdName(strings.TrimSpace(VGScanCmdTmpl))); err != nil {
+		return false, err
+	}
+	// empty stdout is not expected. It must also contain VG name
+	if stdout == "" || !strings.Contains(stdout, name) {
+		return false, errTypes.ErrorNotFound
+	}
+	// find target volume group and check for IO errors
+	if exp, err = regexp.Compile(".*" + name + ".*\n*"); err != nil {
+		return false, err
+	}
+	lines := exp.FindAllString(stderr, -1)
+	for _, line := range lines {
+		if strings.Contains(strings.ToLower(line), ioError) {
+			return true, nil
+		}
+	}
+
+	return false, nil
+}
+
+// VGReactivate inactivates, scans and activates volume group to recover from disk missing scenario
+// Receives name of VG to re-activate
+// Returns error if something went wrong
+func (l *LVM) VGReactivate(name string) error {
+	l.log.Infof("Trying to re-activate volume group %s", name)
+	// re-activate related LVs
+	if _, _, err := l.e.RunCmd(fmt.Sprintf(VGRefreshCmdTmpl, name), command.UseMetrics(true),
+		command.CmdName(strings.TrimSpace(fmt.Sprintf(VGRefreshCmdTmpl, "")))); err != nil {
+		return err
+	}
+
+	return nil
+}
+
 // VGRemove removes volume group, ignore error if VG doesn't exist
 // Receives name of VG to remove
 // Returns error if something went wrong
diff --git a/pkg/eventing/eventing.go b/pkg/eventing/eventing.go
index 91cf8f9..d25864c 100644
--- a/pkg/eventing/eventing.go
+++ b/pkg/eventing/eventing.go
@@ -48,4 +48,9 @@ const (
 
 	FakeAttachInvolved = "FakeAttachInvolved"
 	FakeAttachCleared  = "FakeAttachCleared"
+
+	VolumeGroupScanFailed         = "VolumeGroupScanFailed"
+	VolumeGroupScanInvolved       = "VolumeGroupScanInvolved"
+	VolumeGroupReactivateInvolved = "VolumeGroupReactivateInvolved"
+	VolumeGroupReactivateFailed   = "VolumeGroupReactivateFailed"
 )
diff --git a/pkg/mocks/linuxutils/lvm.go b/pkg/mocks/linuxutils/lvm.go
index fd487dd..53bfad6 100644
--- a/pkg/mocks/linuxutils/lvm.go
+++ b/pkg/mocks/linuxutils/lvm.go
@@ -53,6 +53,20 @@ func (m *MockWrapLVM) VGCreate(name string, pvs ...string) error {
 	return args.Error(0)
 }
 
+// VGScan is a mock implementation
+func (m *MockWrapLVM) VGScan(name string) (bool, error) {
+	args := m.Mock.Called(name)
+
+	return args.Bool(0), args.Error(0)
+}
+
+// VGReactivate is a mock implementation
+func (m *MockWrapLVM) VGReactivate(name string) error {
+	args := m.Mock.Called(name)
+
+	return args.Error(0)
+}
+
 // VGRemove is a mock implementations
 func (m *MockWrapLVM) VGRemove(name string) error {
 	args := m.Mock.Called(name)
diff --git a/pkg/node/volumemgr.go b/pkg/node/volumemgr.go
index 0382ff6..49bc8d6 100644
--- a/pkg/node/volumemgr.go
+++ b/pkg/node/volumemgr.go
@@ -698,7 +698,7 @@ func (m *VolumeManager) updateDrivesCRs(ctx context.Context, drivesFromMgr []*ap
 
 func (m *VolumeManager) handleDriveUpdates(ctx context.Context, updates *driveUpdates) {
 	for _, updDrive := range updates.Updated {
-		m.handleDriveStatusChange(ctx, &updDrive.CurrentState.Spec)
+		m.handleDriveStatusChange(ctx, updDrive)
 	}
 	m.createEventsForDriveUpdates(updates)
 }
@@ -910,22 +910,46 @@ func (m *VolumeManager) getProvisionerForVolume(vol *api.Volume) p.Provisioner {
 // handleDriveStatusChange removes AC that is based on unhealthy drive, returns AC if drive returned to healthy state,
 // mark volumes of the unhealthy drive as unhealthy.
 // Receives golang context and api.Drive that should be handled
-func (m *VolumeManager) handleDriveStatusChange(ctx context.Context, drive *api.Drive) {
+func (m *VolumeManager) handleDriveStatusChange(ctx context.Context, drive updatedDrive) {
+	cur := drive.CurrentState.Spec
+	prev := drive.PreviousState.Spec
 	ll := m.log.WithFields(logrus.Fields{
 		"method":  "handleDriveStatusChange",
-		"driveID": drive.UUID,
+		"driveID": cur.UUID,
 	})
 
-	ll.Infof("The new drive status from DriveMgr is %s", drive.Health)
+	ll.Infof("The new cur status from DriveMgr is %s", cur.Health)
 
 	// Handle resources without LogicalVolumeGroup
 	// Remove AC based on disk with health BAD, SUSPECT, UNKNOWN
-	lvg, err := m.cachedCrHelper.GetLVGByDrive(ctx, drive.UUID)
+	lvg, err := m.cachedCrHelper.GetLVGByDrive(ctx, cur.UUID)
 	if lvg != nil {
+		name := lvg.Name
 		// TODO handle situation when LVG health is changing from Bad/Suspect to Good https://github.com/dell/csi-baremetal/issues/385
-		lvg.Spec.Health = drive.Health
+		lvg.Spec.Health = cur.Health
 		if err := m.k8sClient.UpdateCR(ctx, lvg); err != nil {
-			ll.Errorf("Failed to update lvg CR's %s health status: %v", lvg.Name, err)
+			ll.Errorf("Failed to update lvg CR's %s health status: %v", name, err)
+		}
+		// check for missing disk and re-activate volume group if needed
+		if prev.Status == apiV1.DriveStatusOffline && cur.Status == apiV1.DriveStatusOnline {
+			ll.Infof("Scan volume group %s for IO errors", name)
+			m.recorder.Eventf(lvg, eventing.NormalType, eventing.VolumeGroupScanInvolved, "Check for IO errors")
+			if ok, err := m.lvmOps.VGScan(name); err != nil { //nolint:gocritic
+				ll.Errorf("Failed to scan volume group %s for IO errors: %v", name, err)
+				m.recorder.Eventf(lvg, eventing.ErrorType, eventing.VolumeGroupScanFailed, err.Error())
+			} else if ok {
+				// IO errors detected. Need to re-activate volume group
+				ll.Errorf("IO errors detected for volume group %s", name)
+				m.recorder.Eventf(lvg, eventing.WarningType, eventing.VolumeGroupReactivateInvolved,
+					"IO errors detected")
+				if err := m.lvmOps.VGReactivate(name); err != nil {
+					// need to send an event if operation failed
+					ll.Errorf("Failed to re-activate volume group %s: %v", name, err)
+					m.recorder.Eventf(lvg, eventing.ErrorType, eventing.VolumeGroupReactivateFailed, err.Error())
+				}
+			} else {
+				ll.Infof("No IO errors detected for volume group %s", name)
+			}
 		}
 	} else {
 		errMsg := "Failed get LogicalVolumeGroup CR"
@@ -935,12 +959,12 @@ func (m *VolumeManager) handleDriveStatusChange(ctx context.Context, drive *api.
 		ll.Errorf(errMsg)
 	}
 	// Set disk's health status to volume CR
-	volumes, _ := m.cachedCrHelper.GetVolumesByLocation(ctx, drive.UUID)
+	volumes, _ := m.cachedCrHelper.GetVolumesByLocation(ctx, cur.UUID)
 	for _, vol := range volumes {
-		ll.Infof("Setting updated status %s to volume %s", drive.Health, vol.Name)
+		ll.Infof("Setting updated status %s to volume %s", cur.Health, vol.Name)
 		// save previous health state
 		prevHealthState := vol.Spec.Health
-		vol.Spec.Health = drive.Health
+		vol.Spec.Health = cur.Health
 		// initiate volume release
 		// TODO need to check for specific annotation instead
 		if vol.Spec.Health == apiV1.HealthBad || vol.Spec.Health == apiV1.HealthSuspect {
@@ -956,7 +980,7 @@ func (m *VolumeManager) handleDriveStatusChange(ctx context.Context, drive *api.
 		if vol.Spec.Health == apiV1.HealthBad {
 			m.recorder.Eventf(vol, eventing.WarningType, eventing.VolumeBadHealth,
 				"Volume health transitioned from %s to %s. Inherited from %s drive on %s)",
-				prevHealthState, vol.Spec.Health, drive.Health, drive.NodeId)
+				prevHealthState, vol.Spec.Health, cur.Health, cur.NodeId)
 		}
 	}
 	// Handle resources with LogicalVolumeGroup
diff --git a/variables.mk b/variables.mk
index 64c43ae..e76de17 100644
--- a/variables.mk
+++ b/variables.mk
@@ -9,7 +9,7 @@ EXTENDER_CHART_PATH		:= charts/csi-baremetal-scheduler-extender
 
 ### version
 MAJOR            := 0
-MINOR            := 3
+MINOR            := 4
 PATCH            := 0
 PRODUCT_VERSION  ?= ${MAJOR}.${MINOR}.${PATCH}
 BUILD_REL_A      := $(shell git rev-list HEAD |wc -l)


Test patch:
diff --git a/pkg/base/linuxutils/lvm/lvm_test.go b/pkg/base/linuxutils/lvm/lvm_test.go
index 197ef30..aae64fa 100644
--- a/pkg/base/linuxutils/lvm/lvm_test.go
+++ b/pkg/base/linuxutils/lvm/lvm_test.go
@@ -19,6 +19,7 @@ package lvm
 import (
 	"errors"
 	"fmt"
+	errTypes "github.com/dell/csi-baremetal/pkg/base/error"
 	"strings"
 	"testing"
 
@@ -95,6 +96,57 @@ func TestLinuxUtils_VGCreate(t *testing.T) {
 	err = l.VGCreate(vg, dev1, dev2)
 	assert.Equal(t, expectedErr, err)
 }
+func TestLinuxUtils_VGScan(t *testing.T) {
+	var (
+		e           = &mocks.GoMockExecutor{}
+		l           = NewLVM(e, testLogger)
+		vg          = "test-vg"
+		cmd         = VGScanCmdTmpl
+		ok 			bool
+		err         error
+		expectedErr = errors.New("error")
+	)
+
+	// error not found
+	e.OnCommand(cmd).Return("", "", nil).Times(1)
+	ok, err = l.VGScan(vg)
+	assert.Equal(t, err, errTypes.ErrorNotFound)
+	assert.False(t, ok)
+
+	// error - expected
+	e.OnCommand(cmd).Return("", "", expectedErr).Times(1)
+	ok, err = l.VGScan(vg)
+	assert.False(t, ok)
+	assert.Equal(t, err, expectedErr)
+
+	// IO error detected
+	e.OnCommand(cmd).Return("Found volume group \"" + vg + "\" using metadata type lvm2",
+		"/dev/" + vg + "/test-lv: Input/output error", nil).Times(1)
+	ok, err = l.VGScan(vg)
+	assert.True(t, ok)
+	assert.Nil(t, err)
+
+	// IO error not detected - multiple lines
+	e.OnCommand(cmd).Return("Found volume group \"" + vg + "\" using metadata type lvm2",
+		"/dev/%s/test-lv: no errors\n/dev/other-vg/test-lv: Input/output error", nil).Times(1)
+	ok, err = l.VGScan(vg)
+	assert.False(t, ok)
+	assert.Nil(t, err)
+
+	// IO error detected - multiple lines
+	e.OnCommand(cmd).Return("Found volume group \"" + vg + "\" using metadata type lvm2",
+		"/dev/" + vg + "/test-lv: no errors\n/dev/" + vg + "/test-lv-2: Input/output error", nil).Times(1)
+	ok, err = l.VGScan(vg)
+	assert.True(t, ok)
+	assert.Nil(t, err)
+
+	// error - wrong volume group name
+	incorrectName := "*"
+	e.OnCommand(cmd).Return(incorrectName, "", nil).Times(1)
+	ok, err = l.VGScan(incorrectName)
+	assert.False(t, ok)
+	assert.NotNil(t, err)
+}
 
 func TestLinuxUtils_VGRemove(t *testing.T) {
 	var (
diff --git a/pkg/node/volumemgr_test.go b/pkg/node/volumemgr_test.go
index 09a9742..12d147a 100644
--- a/pkg/node/volumemgr_test.go
+++ b/pkg/node/volumemgr_test.go
@@ -808,16 +808,22 @@ func TestVolumeManager_handleDriveStatusChange(t *testing.T) {
 	drive := drive1
 	drive.UUID = driveUUID
 	drive.Health = apiV1.HealthBad
+	driveCR := &drivecrd.Drive{Spec:drive}
+
+	update := updatedDrive{
+		PreviousState: driveCR,
+		CurrentState:  driveCR,
+	}
 
 	// Check AC deletion
-	vm.handleDriveStatusChange(testCtx, &drive)
+	vm.handleDriveStatusChange(testCtx, update)
 	vol := volCR
 	vol.Spec.Location = driveUUID
 	err = vm.k8sClient.CreateCR(testCtx, testID, &vol)
 	assert.Nil(t, err)
 
 	// Check volume's health change
-	vm.handleDriveStatusChange(testCtx, &drive)
+	vm.handleDriveStatusChange(testCtx, update)
 	rVolume := &vcrd.Volume{}
 	err = vm.k8sClient.ReadCR(testCtx, testID, volCR.Namespace, rVolume)
 	assert.Nil(t, err)
@@ -828,7 +834,7 @@ func TestVolumeManager_handleDriveStatusChange(t *testing.T) {
 	err = vm.k8sClient.CreateCR(testCtx, testLVGName, &lvg)
 	assert.Nil(t, err)
 	// Check lvg's health change
-	vm.handleDriveStatusChange(testCtx, &drive)
+	vm.handleDriveStatusChange(testCtx, update)
 	updatedLVG := &lvgcrd.LogicalVolumeGroup{}
 	err = vm.k8sClient.ReadCR(testCtx, testLVGName, "", updatedLVG)
 	assert.Nil(t, err)