Skill: kaiyuelv/cloud-storage-manager (version 1.0.0)

# cloud-storage-manager - 云存储管理器 SKILL.md for cloud-storage-manager ## Metadata | Field | Value | |-------|-------| | **Name** | cloud-storage-manager | | **Slug** | cloud-storage-manager | | **Version** | 1.0.0 | | **Homepage** | https://github.com/openclaw/cloud-storage-manager | | **Category** | automation | | **Tags** | cloud, storage, s3, oss, cos, aliyun, aws, azure, backup, sync | ## Description ### English Universal cloud storage manager supporting multiple providers (AWS S3, Aliyun OSS, Tencent COS, Azure Blob). Features include file upload/download, bucket management, sync operations, multipart uploads, and CDN integration. ### 中文 通用云存储管理器，支持多种云服务商（AWS S3、阿里云OSS、腾讯云COS、Azure Blob）。功能包括文件上传下载、存储桶管理、同步操作、分片上传和CDN集成。 ## Requirements - Python 3.8+ - boto3 >= 1.26.0 (AWS S3) - aliyun-python-sdk-oss >= 2.17.0 (Aliyun OSS) - qcloud-cos-python-sdk-v5 >= 1.9.0 (Tencent COS) - azure-storage-blob >= 12.14.0 (Azure Blob) ## Configuration ### Environment Variables ```bash # AWS S3 AWS_ACCESS_KEY_ID=your_key AWS_SECRET_ACCESS_KEY=your_secret AWS_REGION=us-east-1 AWS_BUCKET=my-bucket # Aliyun OSS ALIYUN_ACCESS_KEY_ID=your_key [REDACTED_SECRET] ALIYUN_OSS_ENDPOINT=oss-cn-hangzhou.aliyuncs.com ALIYUN_OSS_BUCKET=my-bucket # Tencent COS TENCENT_SECRET_ID=your_id TENCENT_SECRET_KEY=your_key TENCENT_COS_REGION=ap-beijing TENCENT_COS_BUCKET=my-bucket # Azure AZURE_STORAGE_CONNECTION_STRING=your_connection_string AZURE_CONTAINER=my-container ``` ## Usage ### Basic Example ```python from cloud_storage_manager import StorageManager, Provider # Initialize with Aliyun OSS storage = StorageManager(Provider.ALIYUN_OSS) # Upload file storage.upload("local/file.txt", "remote/path/file.txt") # Download file storage.download("remote/path/file.txt", "local/downloaded.txt") # List files files = storage.list_objects(prefix="documents/") # Delete file storage.delete("remote/path/file.txt") # Get signed URL (1 hour expiry) url = storage.get_signed_url("private/file.txt", expires=3600) ``` ### Sync Example ```python from cloud_storage_manager import SyncManager # Sync local directory to cloud sync = SyncManager(storage) sync.sync_to_cloud( local_dir="/path/to/local", remote_prefix="backup/2024/", exclude=["*.tmp", "*.log"], delete_remote=True # Remove files not in local ) # Sync from cloud to local sync.sync_from_cloud( remote_prefix="data/", local_dir="/path/to/download", include=["*.csv", "*.json"] ) ``` ### Multi-Provider Copy ```python # Copy between different providers source = StorageManager(Provider.AWS_S3) dest = StorageManager(Provider.ALIYUN_OSS) # Stream copy without downloading locally from cloud_storage_manager import CrossProviderCopy copier = CrossProviderCopy(source, dest) copier.copy("s3/path/file.zip", "oss/path/file.zip") ``` ## API Reference ### StorageManager - `upload(local_path, remote_path)` - Upload file - `download(remote_path, local_path)` - Download file - `delete(remote_path)` - Delete file - `exists(remote_path)` - Check if file exists - `list_objects(prefix='')` - List files with prefix - `get_size(remote_path)` - Get file size - `get_signed_url(remote_path, expires)` - Get temporary URL - `set_acl(remote_path, acl)` - Set access control ### SyncManager - `sync_to_cloud(local_dir, remote_prefix, **options)` - Upload sync - `sync_from_cloud(remote_prefix, local_dir, **options)` - Download sync - `compare(local_dir, remote_prefix)` - Compare differences ## Examples See `examples/` directory for complete examples. ## Testing ```bash cd /root/.openclaw/workspace/skills/cloud-storage-manager python -m pytest tests/ -v ``` ## License MIT License

path: README.md
content: # Cloud Storage Manager

English | [中文](#中文说明)

## Overview

Universal cloud storage manager supporting AWS S3, Aliyun OSS, Tencent COS, and Azure Blob Storage. Simplifies multi-cloud storage operations.

## Features

- **Multi-Cloud Support**: AWS S3, Aliyun OSS, Tencent COS, Azure Blob
- **Unified API**: Same interface across all providers
- **Sync Operations**: Bidirectional sync between local and cloud
- **Cross-Provider Copy**: Transfer between different cloud providers
- **CDN Integration**: Automatic CDN URL generation
- **Multipart Upload**: Large file support with resume capability

## Installation

```bash
pip install -r requirements.txt
```

## Quick Start

```python
from cloud_storage_manager import StorageManager, Provider

# Initialize Aliyun OSS
storage = StorageManager(Provider.ALIYUN_OSS, {
    "access_key_id": "your_key",
    "access_key_secret": "your_secret",
    "endpoint": "oss-cn-hangzhou.aliyuncs.com",
    "bucket": "my-bucket"
})

# Upload file
storage.upload("local/file.txt", "remote/path/file.txt")

# Generate signed URL
url = storage.get_signed_url("private/file.txt", expires=3600)
print(f"Download URL: {url}")
```

## Supported Providers

| Provider | Service | Region Support |
|----------|---------|----------------|
| AWS | S3 | Global |
| Aliyun | OSS | China + Global |
| Tencent | COS | China + Global |
| Microsoft | Azure Blob | Global |

## License

MIT

---

# 中文说明

## 概述

通用云存储管理器，支持AWS S3、阿里云OSS、腾讯云COS和Azure Blob存储。简化多云存储操作。

## 功能特性

- **多云支持**: AWS S3、阿里云OSS、腾讯云COS、Azure Blob
- **统一API**: 所有提供商使用相同接口
- **同步操作**: 本地与云端双向同步
- **跨云复制**: 在不同云提供商之间传输
- **CDN集成**: 自动生成CDN URL
- **分片上传**: 大文件断点续传支持

## 支持的云服务商

| 服务商 | 服务 | 区域支持 |
|--------|------|----------|
| 亚马逊 | S3 | 全球 |
| 阿里云 | OSS | 中国+全球 |
| 腾讯云 | COS | 中国+全球 |
| 微软 | Azure Blob | 全球 |

## 快速开始

```python
from cloud_storage_manager import StorageManager, Provider

# 初始化阿里云OSS
storage = StorageManager(Provider.ALIYUN_OSS, config={
    "access_key_id": "你的Key",
    "access_key_secret": "你的Secret",
    "endpoint": "oss-cn-hangzhou.aliyuncs.com",
    "bucket": "存储桶名"
})

# 上传文件
storage.upload("本地文件.txt", "云端路径/文件.txt")
```

## 许可证

MIT


path: examples/basic_usage.py
content: #!/usr/bin/env python3
"""
Cloud Storage Manager - Basic Usage Example
云存储管理器 - 基础使用示例

This example demonstrates basic file operations with cloud storage.
本示例展示云存储的基础文件操作。

Note: This example uses mock mode. For real usage, configure credentials.
注意：本示例使用模拟模式。实际使用时需要配置凭证。
"""

import os
import sys

# Add src to path
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))

from cloud_storage_manager import StorageManager, Provider
from cloud_storage_manager.config import load_config


def main():
    """Main example function"""
    print("=" * 60)
    print("Cloud Storage Manager - Basic Usage Example")
    print("云存储管理器 - 基础使用示例")
    print("=" * 60)
    
    # Initialize storage manager (mock mode for demo)
    # 初始化存储管理器（示例使用模拟模式）
    print("\n[1] Initializing storage manager...")
    print("   Provider: Mock Storage (for demonstration)")
    print("   提供商：模拟存储（用于演示）")
    
    # In real usage:
    # storage = StorageManager(Provider.ALIYUN_OSS, {
    #     "access_key_id": "your_key",
    #     "access_key_secret": "your_secret",
    #     "endpoint": "oss-cn-hangzhou.aliyuncs.com",
    #     "bucket": "my-bucket"
    # })
    
    print("✓ Storage manager initialized")
    
    # Simulate operations
    # 模拟操作
    print("\n[2] Simulated Operations / 模拟操作：")
    
    operations = [
        ("Upload file", "upload('local/report.pdf', 'documents/2024/report.pdf')"),
        ("Download file", "download('documents/2024/report.pdf', 'local/downloaded.pdf')"),
        ("List objects", "list_objects(prefix='documents/2024/')"),
        ("Check existence", "exists('documents/2024/report.pdf')"),
        ("Get file size", "get_size('documents/2024/report.pdf')"),
        ("Generate signed URL", "get_signed_url('private/file.txt', expires=3600)"),
        ("Delete file", "delete('temp/old_file.txt')"),
    ]
    
    for op_name, op_code in operations:
        print(f"   ✓ {op_name}")
        print(f"     Code: storage.{op_code}")
    
    # Sync operations
    # 同步操作
    print("\n[3] Sync Operations / 同步操作：")
    
    sync_examples = [
        ("Sync to cloud", "sync.sync_to_cloud('/local/data', 'backup/2024/')"),
        ("Sync from cloud", "sync.sync_from_cloud('data/export/', '/local/download')"),
        ("Compare directories", "sync.compare('/local/data', 'backup/2024/')"),
    ]
    
    for op_name, op_code in sync_examples:
        print(f"   ✓ {op_name}")
        print(f"     Code: {op_code}")
    
    # Multi-provider copy
    # 多云复制
    print("\n[4] Cross-Provider Copy / 跨云复制：")
    print("   # Copy from AWS S3 to Aliyun OSS")
    print("   source = StorageManager(Provider.AWS_S3, aws_config)")
    print("   dest = StorageManager(Provider.ALIYUN_OSS, oss_config)")
    print("   copier = CrossProviderCopy(source, dest)")
    print("   copier.copy('s3/file.zip', 'oss/file.zip')")
    
    # Configuration example
    # 配置示例
    print("\n[5] Configuration Example / 配置示例：")
    print("""
    # .env file / 环境变量文件
    ALIYUN_ACCESS_KEY_ID=your_access_key
    [REDACTED_SECRET]
    ALIYUN_OSS_ENDPOINT=oss-cn-hangzhou.aliyuncs.com
    ALIYUN_OSS_BUCKET=my-bucket
    
    # Or initialize from environment / 或从环境初始化
    config = load_config('aliyun_oss')
    storage = StorageManager(Provider.ALIYUN_OSS, config)
    """)
    
    print("\n" + "=" * 60)
    print("Example completed!")
    print("For real usage, configure your cloud provider credentials.")
    print("示例完成！实际使用时请配置云服务商凭证。")
    print("=" * 60)


if __name__ == "__main__":
    main()


path: requirements.txt
content: # Cloud Storage Manager Requirements
# 云存储管理器依赖

# AWS S3 Support - AWS S3支持
boto3>=1.26.0                  # AWS SDK for Python
botocore>=1.29.0               # AWS core library

# Aliyun OSS Support - 阿里云OSS支持
aliyun-python-sdk-core>=2.13.0 # Aliyun core SDK
aliyun-python-sdk-oss>=2.17.0  # Aliyun OSS SDK
oss2>=2.17.0                   # Aliyun OSS Python SDK

# Tencent COS Support - 腾讯云COS支持
qcloud-cos-python-sdk-v5>=1.9.0 # Tencent COS SDK

# Azure Blob Support - Azure Blob支持
azure-storage-blob>=12.14.0    # Azure Blob SDK
azure-identity>=1.12.0         # Azure authentication

# Utilities - 工具库
tqdm>=4.65.0                   # Progress bars
python-dotenv>=1.0.0           # Environment variables
pydantic>=2.0.0                # Data validation
pytest>=7.0.0                  # Testing framework
pytest-asyncio>=0.21.0         # Async testing


path: src/cloud_storage_manager/__init__.py
content: """
Cloud Storage Manager - Universal cloud storage management
云存储管理器 - 通用云存储管理

Features:
- Multi-cloud support (AWS S3, Aliyun OSS, Tencent COS, Azure Blob)
- Unified API across all providers
- Sync operations
- Cross-provider copy
"""

__version__ = "1.0.0"
__author__ = "OpenClaw"

from .storage import StorageManager, Provider
from .sync import SyncManager, CrossProviderCopy

__all__ = [
    "StorageManager",
    "Provider",
    "SyncManager",
    "CrossProviderCopy",
]


path: src/cloud_storage_manager/config.py
content: """
Configuration module for Cloud Storage Manager
云存储管理器配置模块
"""

import os
from typing import Dict, Any


def load_config(provider: str) -> Dict[str, Any]:
    """
    Load configuration from environment variables
    从环境变量加载配置
    
    Args:
        provider: Provider name (e.g., 'aliyun_oss', 'aws_s3')
        
    Returns:
        Configuration dictionary
    """
    config = {}
    
    provider_configs = {
        'aliyun_oss': {
            'access_key_id': 'ALIYUN_ACCESS_KEY_ID',
            'access_key_secret': 'ALIYUN_ACCESS_KEY_SECRET',
            'endpoint': 'ALIYUN_OSS_ENDPOINT',
            'bucket': 'ALIYUN_OSS_BUCKET',
        },
        'aws_s3': {
            'access_key_id': 'AWS_ACCESS_KEY_ID',
            'secret_access_key': 'AWS_SECRET_ACCESS_KEY',
            'region': 'AWS_REGION',
            'bucket': 'AWS_BUCKET',
        },
        'tencent_cos': {
            'secret_id': 'TENCENT_SECRET_ID',
            'secret_key': 'TENCENT_SECRET_KEY',
            'region': 'TENCENT_COS_REGION',
            'bucket': 'TENCENT_COS_BUCKET',
        },
        'azure_blob': {
            'connection_string': 'AZURE_STORAGE_CONNECTION_STRING',
            'container': 'AZURE_CONTAINER',
        },
    }
    
    if provider not in provider_configs:
        raise ValueError(f"Unknown provider: {provider}")
    
    for key, env_var in provider_configs[provider].items():
        value = os.getenv(env_var)
        if value:
            config[key] = value
    
    return config


path: tests/test_storage.py
content: #!/usr/bin/env python3
"""
Cloud Storage Manager - Unit Tests
云存储管理器 - 单元测试

Run tests: python -m pytest tests/test_storage.py -v
运行测试: python -m pytest tests/test_storage.py -v
"""

import os
import sys
import unittest
import tempfile
import hashlib
from datetime import datetime, timedelta

# Add src to path
sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', 'src'))


class MockStorageManager:
    """Mock storage manager for testing"""
    
    def __init__(self, provider, config=None):
        self.provider = provider
        self.config = config or {}
        self.files = {}  # Simulate storage
        
    def upload(self, local_path, remote_path):
        """Mock upload"""
        content = f"Mock content from {local_path}"
        self.files[remote_path] = {
            'content': content,
            'size': len(content),
            'created_at': datetime.now()
        }
        return {'success': True, 'path': remote_path}
    
    def download(self, remote_path, local_path):
        """Mock download"""
        if remote_path not in self.files:
            raise FileNotFoundError(f"File not found: {remote_path}")
        return {'success': True, 'size': self.files[remote_path]['size']}
    
    def delete(self, remote_path):
        """Mock delete"""
        if remote_path in self.files:
            del self.files[remote_path]
            return True
        return False
    
    def exists(self, remote_path):
        """Check if file exists"""
        return remote_path in self.files
    
    def list_objects(self, prefix=''):
        """List objects with prefix"""
        return [
            {'key': k, 'size': v['size'], 'modified': v['created_at']}
            for k, v in self.files.items()
            if k.startswith(prefix)
        ]
    
    def get_size(self, remote_path):
        """Get file size"""
        if remote_path in self.files:
            return self.files[remote_path]['size']
        raise FileNotFoundError(f"File not found: {remote_path}")
    
    def get_signed_url(self, remote_path, expires=3600):
        """Generate signed URL"""
        if remote_path not in self.files:
            raise FileNotFoundError(f"File not found: {remote_path}")
        expiry = datetime.now() + timedelta(seconds=expires)
        return f"https://mock-storage.example.com/{remote_path}?expires={expiry.timestamp()}"


class MockSyncManager:
    """Mock sync manager"""
    
    def __init__(self, storage):
        self.storage = storage
        self.sync_history = []
    
    def sync_to_cloud(self, local_dir, remote_prefix, exclude=None, delete_remote=False):
        """Mock sync to cloud"""
        self.sync_history.append({
            'direction': 'to_cloud',
            'local': local_dir,
            'remote': remote_prefix
        })
        return {'uploaded': 5, 'skipped': 2, 'deleted': 0 if not delete_remote else 1}
    
    def sync_from_cloud(self, remote_prefix, local_dir, include=None):
        """Mock sync from cloud"""
        self.sync_history.append({
            'direction': 'from_cloud',
            'remote': remote_prefix,
            'local': local_dir
        })
        return {'downloaded': 3, 'skipped': 1}
    
    def compare(self, local_dir, remote_prefix):
        """Compare directories"""
        return {
            'local_only': ['file1.txt'],
            'remote_only': ['file2.txt'],
            'different': ['file3.txt'],
            'same': ['file4.txt']
        }


class TestCloudStorageManager(unittest.TestCase):
    """Test cases for cloud storage manager"""
    
    def setUp(self):
        """Set up test storage"""
        self.storage = MockStorageManager("mock")
    
    def test_upload(self):
        """Test upload operation"""
        result = self.storage.upload("local/test.txt", "remote/test.txt")
        self.assertTrue(result['success'])
        self.assertEqual(result['path'], "remote/test.txt")
        self.assertIn("remote/test.txt", self.storage.files)
    
    def test_download(self):
        """Test download operation"""
        # First upload
        self.storage.upload("local/test.txt", "remote/test.txt")
        
        # Then download
        result = self.storage.download("remote/test.txt", "local/downloaded.txt")
        self.assertTrue(result['success'])
    
    def test_download_not_found(self):
        """Test download non-existent file"""
        with self.assertRaises(FileNotFoundError):
            self.storage.download("remote/nonexistent.txt", "local.txt")
    
    def test_delete(self):
        """Test delete operation"""
        self.storage.upload("local/test.txt", "remote/test.txt")
        
        result = self.storage.delete("remote/test.txt")
        self.assertTrue(result)
        self.assertNotIn("remote/test.txt", self.storage.files)
    
    def test_delete_not_found(self):
        """Test delete non-existent file"""
        result = self.storage.delete("remote/nonexistent.txt")
        self.assertFalse(result)
    
    def test_exists(self):
        """Test exists check"""
        self.assertFalse(self.storage.exists("remote/test.txt"))
        
        self.storage.upload("local/test.txt", "remote/test.txt")
        self.assertTrue(self.storage.exists("remote/test.txt"))
    
    def test_list_objects(self):
        """Test list objects"""
        # Upload files with different prefixes
        files = [
            ("local/a.txt", "documents/2024/a.txt"),
            ("local/b.txt", "documents/2024/b.txt"),
            ("local/c.txt", "images/photo.jpg"),
        ]
        for local, remote in files:
            self.storage.upload(local, remote)
        
        # List with prefix
        docs = self.storage.list_objects(prefix="documents/")
        self.assertEqual(len(docs), 2)
        
        # List all
        all_files = self.storage.list_objects()
        self.assertEqual(len(all_files), 3)
    
    def test_get_size(self):
        """Test get file size"""
        self.storage.upload("local/test.txt", "remote/test.txt")
        
        size = self.storage.get_size("remote/test.txt")
        self.assertGreater(size, 0)
    
    def test_get_size_not_found(self):
        """Test get size of non-existent file"""
        with self.assertRaises(FileNotFoundError):
            self.storage.get_size("remote/nonexistent.txt")
    
    def test_get_signed_url(self):
        """Test signed URL generation"""
        self.storage.upload("local/test.txt", "remote/test.txt")
        
        url = self.storage.get_signed_url("remote/test.txt", expires=3600)
        self.assertIn("https://mock-storage.example.com/", url)
        self.assertIn("expires=", url)
    
    def test_get_signed_url_not_found(self):
        """Test signed URL for non-existent file"""
        with self.assertRaises(FileNotFoundError):
            self.storage.get_signed_url("remote/nonexistent.txt")


class TestSyncManager(unittest.TestCase):
    """Test cases for sync manager"""
    
    def setUp(self):
        """Set up test"""
        self.storage = MockStorageManager("mock")
        self.sync = MockSyncManager(self.storage)
    
    def test_sync_to_cloud(self):
        """Test sync to cloud"""
        result = self.sync.sync_to_cloud("/local/data", "backup/2024/")
        
        self.assertEqual(result['uploaded'], 5)
        self.assertEqual(result['skipped'], 2)
        self.assertEqual(self.sync.sync_history[0]['direction'], 'to_cloud')
    
    def test_sync_from_cloud(self):
        """Test sync from cloud"""
        result = self.sync.sync_from_cloud("data/", "/local/download")
        
        self.assertEqual(result['downloaded'], 3)
        self.assertEqual(self.sync.sync_history[0]['direction'], 'from_cloud')
    
    def test_compare(self):
        """Test directory comparison"""
        result = self.sync.compare("/local", "remote/")
        
        self.assertIn('local_only', result)
        self.assertIn('remote_only', result)
        self.assertIn('different', result)
        self.assertIn('same', result)


class TestProviders(unittest.TestCase):
    """Test provider constants"""
    
    def test_provider_names(self):
        """Test provider name consistency"""
        providers = ["AWS_S3", "ALIYUN_OSS", "TENCENT_COS", "AZURE_BLOB"]
        for provider in providers:
            # Just verify the name format
            self.assertTrue(provider.isupper())
            self.assertIn("_", provider)


class TestEdgeCases(unittest.TestCase):
    """Test edge cases"""
    
    def test_empty_storage(self):
        """Test operations on empty storage"""
        storage = MockStorageManager("mock")
        
        # List should return empty
        files = storage.list_objects()
        self.assertEqual(len(files), 0)
        
        # Exists should return False
        self.assertFalse(storage.exists("anything.txt"))
    
    def test_special_characters_in_path(self):
        """Test paths with special characters"""
        storage = MockStorageManager("mock")
        
        special_paths = [
            "path with spaces/file.txt",
            "path-with-dashes/file.txt",
            "path_with_underscores/file.txt",
            "2024/01/15/file.txt",
        ]
        
        for i, path in enumerate(special_paths):
            storage.upload(f"local{i}.txt", path)
            self.assertTrue(storage.exists(path))


if __name__ == '__main__':
    unittest.main()


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)