Pythonista3 でAVAudioSourceNode を使って音を生成して鳴らそう！ 後編

この記事は、[Pythonista3 Advent Calendar 2022](https://qiita.com/advent-calendar/2022/pythonista3) の16日目の記事です。

https://qiita.com/advent-calendar/2022/pythonista3


一方的な偏った目線で、Pythonista3 を紹介していきます。

ほぼ毎日iPhone（Pythonista3）で、コーディングをしている者です。よろしくお願いします。

以下、私の2022年12月時点の環境です。

```sysInfo.log
--- SYSTEM INFORMATION ---
* Pythonista 3.3 (330025), Default interpreter 3.6.1
* iOS 16.1.1, model iPhone12,1, resolution (portrait) 828.0 x 1792.0 @ 2.0
```

他の環境(iPad や端末の種類、iOS のバージョン違い)では、意図としない挙動(エラーになる)なる場合もあります。ご了承ください。

ちなみに、`model iPhone12,1` は、iPhone11 です。

## 前回までのあらすじ

- 音を出せた
  - `objc_util` モジュールで`AVAudioEngine` を中心に実装
    - `AVAudioSourceNode`
      - `ObjCBlock` で、音声データのルーティング
        - 数式を使ったsine波の生成
      - `AudioBuffer`, `AudioBufferList` 構造体の作成
    - connect によるNode の連結
- 音を止められた
  - `ui` モジュールでアプリライク & 終了指示の連携

まとめると、こんなにさっぱりするのですね！

今回は、波形の種類を変えたり、可視化したり、インタラクティブに操作できるようにしていきましょう！！

`objc_util` 色は弱めでPython （Pythonista3）の処理が中心です。

```前回までのコード.py
from math import sin, pi
import ctypes

from objc_util import ObjCClass, ObjCBlock
import ui

import pdbg

CHANNEL = 1

OSStatus = ctypes.c_int32

AVAudioEngine = ObjCClass('AVAudioEngine')
AVAudioSourceNode = ObjCClass('AVAudioSourceNode')
AVAudioFormat = ObjCClass('AVAudioFormat')


class AudioBuffer(ctypes.Structure):
  _fields_ = [
    ('mNumberChannels', ctypes.c_uint32),
    ('mDataByteSize', ctypes.c_uint32),
    ('mData', ctypes.c_void_p),
  ]


class AudioBufferList(ctypes.Structure):
  _fields_ = [
    ('mNumberBuffers', ctypes.c_uint32),
    ('mBuffers', AudioBuffer * CHANNEL),
  ]


class Synth:
  def __init__(self):
    self.audioEngine: AVAudioEngine
    self.sampleRate: float = 44100.0  # set_up メソッド: outputNode より確定
    self.deltaTime: float = 0.0  # 1/sampleRate 時間間隔
    self.timex: float = 0.0  # Render の間隔カウンター

    self.render_block = ObjCBlock(
      self.source_node_render,
      restype=OSStatus,
      argtypes=[
        ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p,
        ctypes.POINTER(AudioBufferList)
      ])

    self.set_up()

  def set_up(self):
    audioEngine = AVAudioEngine.new()
    sourceNode = AVAudioSourceNode.alloc()
    mainMixer = audioEngine.mainMixerNode()
    outputNode = audioEngine.outputNode()
    format = outputNode.inputFormatForBus_(0)

    self.sampleRate = format.sampleRate()
    self.deltaTime = 1 / self.sampleRate

    inputFormat = AVAudioFormat.alloc(
    ).initWithCommonFormat_sampleRate_channels_interleaved_(
      format.commonFormat(), self.sampleRate, CHANNEL, format.isInterleaved())

    sourceNode.initWithFormat_renderBlock_(inputFormat, self.render_block)

    audioEngine.attachNode_(sourceNode)
    sourceNode.volume = 0.5

    audioEngine.connect_to_format_(sourceNode, mainMixer, inputFormat)
    audioEngine.connect_to_format_(mainMixer, outputNode, inputFormat)

    audioEngine.prepare()
    self.audioEngine = audioEngine

  def source_node_render(self,
                         _cmd: ctypes.c_void_p,
                         _isSilence_ptr: ctypes.c_void_p,
                         _timestamp_ptr: ctypes.c_void_p,
                         frameCount: ctypes.c_void_p,
                         outputData_ptr: ctypes.POINTER) -> OSStatus:
    ablPointer = outputData_ptr.contents
    for frame in range(frameCount):
      # 出力波形を変えていく
      sampleVal = sin(440.0 * 2.0 * pi * self.timex)
      self.timex += self.deltaTime

      for bufferr in range(ablPointer.mNumberBuffers):
        _mData = ablPointer.mBuffers[bufferr].mData
        _pointer = ctypes.POINTER(ctypes.c_float * frameCount)
        buffer = ctypes.cast(_mData, _pointer).contents
        buffer[frame] = sampleVal
    return 0

  def start(self):
    self.audioEngine.startAndReturnError_(None)

  def stop(self):
    self.audioEngine.stop()


class View(ui.View):
  def __init__(self, *args, **kwargs):
    ui.View.__init__(self, *args, **kwargs)
    self.synth = Synth()
    self.synth.start()

  def will_close(self):
    self.synth.stop()


if __name__ == '__main__':
  view = View()
  view.present()
  #view.present(style='fullscreen', orientations=['portrait'])

```

## オシレーターをつくる

Oscillator（発振器・OSC）は、シンセサイザーで基本波形を作り出す機能です。

- 正弦波: sine
- 三角波: triangle
- ノコギリ波: sawtooth
- 矩形波: square

[Building a Synthesizer in Swift. Making audio waveforms with… | by SwiftMoji | Better Programming](https://betterprogramming.pub/building-a-synthesizer-in-swift-866cd15b731)

https://betterprogramming.pub/building-a-synthesizer-in-swift-866cd15b731


こちらを参考に作っていきます。

### White Noise

おーっと、早速OSC で紹介していないのがでてきましたね😇

面白いですし、先々にやってしまった方がいいと思いまして、先に実装します。

```whiteNoise.py
from random import uniform

# --- sine
#sampleVal = sin(440.0 * 2.0 * pi * self.timex)
# --- whiteNoise 
sampleVal = uniform(-1.0, 1.0)
```

`random.random()` ですと、0.0以上1.0未満 の抽出です。

音声データとして`-1.0 〜 1.0` が欲しいので、`random.uniform(-1.0, 1.0)` で範囲を指定しています。

砂嵐の音になりましたね（現代人は、テレビやラジオのノイズ音とか知らないんだろうな）。

White Noise 以降は、関数化して作成していきます。

### 正弦波: sine

すでに、前回で呼び出し確認ができてる`sine` ですが、関数化をしていきます。

`amplitude`（振幅）と`frequency`（周波数）を変数として事前に定義しておきます。

（振幅や周波数等の説明に関してはここでは割愛させていただきます）

```py
# --- OSC
amplitude: float = 1.0
frequency: float = 440.0


def sine(time):
  wave = amplitude * sin(2.0 * pi * frequency * time)
  return wave

```

全体像:

```sineを追加.py
from math import sin, pi
from random import uniform
import ctypes

from objc_util import ObjCClass, ObjCBlock
import ui

import pdbg

CHANNEL = 1

OSStatus = ctypes.c_int32

AVAudioEngine = ObjCClass('AVAudioEngine')
AVAudioSourceNode = ObjCClass('AVAudioSourceNode')
AVAudioFormat = ObjCClass('AVAudioFormat')


class AudioBuffer(ctypes.Structure):
  _fields_ = [
    ('mNumberChannels', ctypes.c_uint32),
    ('mDataByteSize', ctypes.c_uint32),
    ('mData', ctypes.c_void_p),
  ]


class AudioBufferList(ctypes.Structure):
  _fields_ = [
    ('mNumberBuffers', ctypes.c_uint32),
    ('mBuffers', AudioBuffer * CHANNEL),
  ]


# --- OSC
amplitude: float = 1.0
frequency: float = 440.0


def white_noise():
  return uniform(-1.0, 1.0)


def sine(time):
  wave = amplitude * sin(2.0 * pi * frequency * time)
  return wave


class Synth:
  def __init__(self):
    self.audioEngine: AVAudioEngine
    self.sampleRate: float = 44100.0  # set_up メソッド: outputNode より確定
    self.deltaTime: float = 0.0  # 1/sampleRate 時間間隔
    self.timex: float = 0.0  # Render の間隔カウンター

    self.render_block = ObjCBlock(
      self.source_node_render,
      restype=OSStatus,
      argtypes=[
        ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p,
        ctypes.POINTER(AudioBufferList)
      ])

    self.set_up()

  def set_up(self):
    audioEngine = AVAudioEngine.new()
    sourceNode = AVAudioSourceNode.alloc()
    mainMixer = audioEngine.mainMixerNode()
    outputNode = audioEngine.outputNode()
    format = outputNode.inputFormatForBus_(0)

    self.sampleRate = format.sampleRate()
    self.deltaTime = 1 / self.sampleRate

    inputFormat = AVAudioFormat.alloc(
    ).initWithCommonFormat_sampleRate_channels_interleaved_(
      format.commonFormat(), self.sampleRate, CHANNEL, format.isInterleaved())

    sourceNode.initWithFormat_renderBlock_(inputFormat, self.render_block)

    audioEngine.attachNode_(sourceNode)
    sourceNode.volume = 0.2

    audioEngine.connect_to_format_(sourceNode, mainMixer, inputFormat)
    audioEngine.connect_to_format_(mainMixer, outputNode, inputFormat)

    audioEngine.prepare()
    self.audioEngine = audioEngine

  def source_node_render(self,
                         _cmd: ctypes.c_void_p,
                         _isSilence_ptr: ctypes.c_void_p,
                         _timestamp_ptr: ctypes.c_void_p,
                         frameCount: ctypes.c_void_p,
                         outputData_ptr: ctypes.POINTER) -> OSStatus:
    ablPointer = outputData_ptr.contents
    for frame in range(frameCount):
      # 出力波形を変えていく
      # --- sine
      #sampleVal = sin(440.0 * 2.0 * pi * self.timex)
      # --- whiteNoise
      #sampleVal = uniform(-1.0, 1.0)
      #sampleVal = white_noise()
      # --- sine
      sampleVal = sine(self.timex)

      self.timex += self.deltaTime

      for bufferr in range(ablPointer.mNumberBuffers):
        _mData = ablPointer.mBuffers[bufferr].mData
        _pointer = ctypes.POINTER(ctypes.c_float * frameCount)
        buffer = ctypes.cast(_mData, _pointer).contents
        buffer[frame] = sampleVal
    return 0

  def start(self):
    self.audioEngine.startAndReturnError_(None)

  def stop(self):
    self.audioEngine.stop()


class View(ui.View):
  def __init__(self, *args, **kwargs):
    ui.View.__init__(self, *args, **kwargs)
    self.synth = Synth()
    self.synth.start()

  def will_close(self):
    self.synth.stop()


if __name__ == '__main__':
  view = View()
  view.present()
  #view.present(style='fullscreen', orientations=['portrait'])

```

他の波の関数は、`sine` の下に追加していきます。

さっくり関数のみ紹介していきます（この後class 化もしていくので面倒なら見てるだけでもいいかもしれません）。

### 三角波: triangle

```py
def triangle(time):
  period = 1.0 / frequency
  currentTime = time % period
  value = currentTime / period
  result = 0.0
  if value < 0.25:
    result = value * 4
  elif value < 0.75:
    result = 2.0 - (value * 4.0)
  else:
    result = value * 4 - 4.0
  wave = amplitude * result
  return wave
```

### ノコギリ波: sawtooth

```py
def sawtooth(time):
  period = 1.0 / frequency
  currentTime = time % period
  wave = amplitude * ((currentTime / period) * 2 - 1.0)
  return wave
```

### 矩形波: square

```py
def square(time):
  period = 1.0 / frequency
  currentTime = time % period
  if (currentTime / period) < 0.5:
    wave = amplitude
  else:
    wave = -1.0 * amplitude
  return wave
```

### class 化と、インタラクティブな拡張性

```関数実装.py
from math import sin, pi
from random import uniform
import ctypes

from objc_util import ObjCClass, ObjCBlock
import ui

import pdbg

CHANNEL = 1

OSStatus = ctypes.c_int32

AVAudioEngine = ObjCClass('AVAudioEngine')
AVAudioSourceNode = ObjCClass('AVAudioSourceNode')
AVAudioFormat = ObjCClass('AVAudioFormat')


class AudioBuffer(ctypes.Structure):
  _fields_ = [
    ('mNumberChannels', ctypes.c_uint32),
    ('mDataByteSize', ctypes.c_uint32),
    ('mData', ctypes.c_void_p),
  ]


class AudioBufferList(ctypes.Structure):
  _fields_ = [
    ('mNumberBuffers', ctypes.c_uint32),
    ('mBuffers', AudioBuffer * CHANNEL),
  ]


# --- OSC
amplitude: float = 1.0
frequency: float = 440.0


def white_noise():
  return uniform(-1.0, 1.0)


def sine(time):
  wave = amplitude * sin(2.0 * pi * frequency * time)
  return wave


def triangle(time):
  period = 1.0 / frequency
  currentTime = time % period
  value = currentTime / period
  result = 0.0
  if value < 0.25:
    result = value * 4
  elif value < 0.75:
    result = 2.0 - (value * 4.0)
  else:
    result = value * 4 - 4.0
  wave = amplitude * result
  return wave


def sawtooth(time):
  period = 1.0 / frequency
  currentTime = time % period
  wave = amplitude * ((currentTime / period) * 2 - 1.0)
  return wave


def square(time):
  period = 1.0 / frequency
  currentTime = time % period
  if (currentTime / period) < 0.5:
    wave = amplitude
  else:
    wave = -1.0 * amplitude
  return wave


class Synth:
  def __init__(self):
    self.audioEngine: AVAudioEngine
    self.sampleRate: float = 44100.0  # set_up メソッド: outputNode より確定
    self.deltaTime: float = 0.0  # 1/sampleRate 時間間隔
    self.timex: float = 0.0  # Render の間隔カウンター

    self.render_block = ObjCBlock(
      self.source_node_render,
      restype=OSStatus,
      argtypes=[
        ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p,
        ctypes.POINTER(AudioBufferList)
      ])

    self.set_up()

  def set_up(self):
    audioEngine = AVAudioEngine.new()
    sourceNode = AVAudioSourceNode.alloc()
    mainMixer = audioEngine.mainMixerNode()
    outputNode = audioEngine.outputNode()
    format = outputNode.inputFormatForBus_(0)

    self.sampleRate = format.sampleRate()
    self.deltaTime = 1 / self.sampleRate

    inputFormat = AVAudioFormat.alloc(
    ).initWithCommonFormat_sampleRate_channels_interleaved_(
      format.commonFormat(), self.sampleRate, CHANNEL, format.isInterleaved())

    sourceNode.initWithFormat_renderBlock_(inputFormat, self.render_block)

    audioEngine.attachNode_(sourceNode)
    sourceNode.volume = 0.2

    audioEngine.connect_to_format_(sourceNode, mainMixer, inputFormat)
    audioEngine.connect_to_format_(mainMixer, outputNode, inputFormat)

    audioEngine.prepare()
    self.audioEngine = audioEngine

  def source_node_render(self,
                         _cmd: ctypes.c_void_p,
                         _isSilence_ptr: ctypes.c_void_p,
                         _timestamp_ptr: ctypes.c_void_p,
                         frameCount: ctypes.c_void_p,
                         outputData_ptr: ctypes.POINTER) -> OSStatus:
    ablPointer = outputData_ptr.contents
    for frame in range(frameCount):
      # 出力波形を変えていく
      # --- whiteNoise
      #sampleVal = uniform(-1.0, 1.0)
      #sampleVal = white_noise()
      # --- sine
      #sampleVal = sin(440.0 * 2.0 * pi * self.timex)
      #sampleVal = sine(self.timex)
      # --- triangle
      #sampleVal = triangle(self.timex)
      # --- sawtooth
      #sampleVal = sawtooth(self.timex)
      # --- square
      sampleVal = square(self.timex)

      self.timex += self.deltaTime

      for bufferr in range(ablPointer.mNumberBuffers):
        _mData = ablPointer.mBuffers[bufferr].mData
        _pointer = ctypes.POINTER(ctypes.c_float * frameCount)
        buffer = ctypes.cast(_mData, _pointer).contents
        buffer[frame] = sampleVal
    return 0

  def start(self):
    self.audioEngine.startAndReturnError_(None)

  def stop(self):
    self.audioEngine.stop()


class View(ui.View):
  def __init__(self, *args, **kwargs):
    ui.View.__init__(self, *args, **kwargs)
    self.synth = Synth()
    self.synth.start()

  def will_close(self):
    self.synth.stop()


if __name__ == '__main__':
  view = View()
  view.present()
  #view.present(style='fullscreen', orientations=['portrait'])

```

一通り実装が完了しました。が、`Oscillator` としてclass 化した方が都合が良さそうですね。

いまは`Synth` のRender メソッド内で、毎回書き換え（コメントアウト）て、それぞれの波を選んでいるので、確認のイテレートも面倒です。

#### `class Oscillator:`

ひとまとめにしました。

`self.wave_types` の配列で波の関数をindex で呼び出せるようにしています。

また、`white_noise` メソッドは、引数が不要ですがメソッド全体と整える必要があるため、`_` としています。

```classの部分のみ.py
class Oscillator:
  def __init__(self):
    self.amplitude: float = 1.0
    self.frequency: float = 440.0
    self.wave_types = [
      self.sine,
      self.triangle,
      self.sawtooth,
      self.square,
      self.white_noise,
    ]

  def sine(self, time):
    wave = self.amplitude * sin(2.0 * pi * self.frequency * time)
    return wave

  def triangle(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    value = currentTime / period
    result = 0.0
    if value < 0.25:
      result = value * 4
    elif value < 0.75:
      result = 2.0 - (value * 4.0)
    else:
      result = value * 4 - 4.0
    wave = self.amplitude * result
    return wave

  def sawtooth(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    wave = self.amplitude * ((currentTime / period) * 2 - 1.0)
    return wave

  def square(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    if (currentTime / period) < 0.5:
      wave = self.amplitude
    else:
      wave = -1.0 * self.amplitude
    return wave

  def white_noise(self, _):
    return uniform(-1.0, 1.0)
```

#### インタラクティブに

実行時に変化させるように、class `View` としている`ui.View` の中に、スライダーで選択できるように組み込んでいきます。

##### osc 選択

```py
def setup_type_slider(self):
  self.type_len = len(self.osc.wave_types) - 1
  self.type_osc = ui.Slider()
  self.type_osc.continuous = False
  self.type_osc.value = 0
  self.type_osc.flex = 'W'
  self.type_osc.action = self.change_osc
  self.add_subview(self.type_osc)
  
def change_osc(self, sender):
  val = int(sender.value * self.type_len)
  self.toneGenerator = self.osc.wave_types[val]
  self.type_osc.value = val / self.type_len
  print(self.toneGenerator)  # ← 今回の確認用
```

`print` を仕込んでいるので、ocs の切り替わりで現在の波を出力しています。

![img221205_231256.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2953777/8f2e2b20-f863-6e5c-14e7-18b025ffc5db.gif)


`ui.Slider.continuous = False` にて、（移動時ではなく）スライダーの位置がfix 時に数値を送り出しています。

`val = int(sender.value * self.type_len)` にて、配列index を指定。

`self.type_osc.value = val / self.type_len` にて、`ui.Slider.value` の位置調整をいています。

```slider実装.py
from math import sin, pi
from random import uniform
import ctypes

from objc_util import ObjCClass, ObjCBlock
import ui

import pdbg

CHANNEL = 1

OSStatus = ctypes.c_int32

AVAudioEngine = ObjCClass('AVAudioEngine')
AVAudioSourceNode = ObjCClass('AVAudioSourceNode')
AVAudioFormat = ObjCClass('AVAudioFormat')


class AudioBuffer(ctypes.Structure):
  _fields_ = [
    ('mNumberChannels', ctypes.c_uint32),
    ('mDataByteSize', ctypes.c_uint32),
    ('mData', ctypes.c_void_p),
  ]


class AudioBufferList(ctypes.Structure):
  _fields_ = [
    ('mNumberBuffers', ctypes.c_uint32),
    ('mBuffers', AudioBuffer * CHANNEL),
  ]


# --- OSC
class Oscillator:
  def __init__(self):
    self.amplitude: float = 1.0
    self.frequency: float = 440.0
    self.wave_types = [
      self.sine,
      self.triangle,
      self.sawtooth,
      self.square,
      self.white_noise,
    ]

  def sine(self, time):
    wave = self.amplitude * sin(2.0 * pi * self.frequency * time)
    return wave

  def triangle(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    value = currentTime / period
    result = 0.0
    if value < 0.25:
      result = value * 4
    elif value < 0.75:
      result = 2.0 - (value * 4.0)
    else:
      result = value * 4 - 4.0
    wave = self.amplitude * result
    return wave

  def sawtooth(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    wave = self.amplitude * ((currentTime / period) * 2 - 1.0)
    return wave

  def square(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    if (currentTime / period) < 0.5:
      wave = self.amplitude
    else:
      wave = -1.0 * self.amplitude
    return wave

  def white_noise(self, _):
    return uniform(-1.0, 1.0)


class Synth:
  def __init__(self, parent):
    self.parent: ui.View = parent
    self.audioEngine: AVAudioEngine
    self.sampleRate: float = 44100.0  # set_up メソッド: outputNode より確定
    self.deltaTime: float = 0.0  # 1/sampleRate 時間間隔
    self.timex: float = 0.0  # Render の間隔カウンター

    self.render_block = ObjCBlock(
      self.source_node_render,
      restype=OSStatus,
      argtypes=[
        ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p,
        ctypes.POINTER(AudioBufferList)
      ])

    self.set_up()

  def set_up(self):
    audioEngine = AVAudioEngine.new()
    sourceNode = AVAudioSourceNode.alloc()
    mainMixer = audioEngine.mainMixerNode()
    outputNode = audioEngine.outputNode()
    format = outputNode.inputFormatForBus_(0)

    self.sampleRate = format.sampleRate()
    self.deltaTime = 1 / self.sampleRate

    inputFormat = AVAudioFormat.alloc(
    ).initWithCommonFormat_sampleRate_channels_interleaved_(
      format.commonFormat(), self.sampleRate, CHANNEL, format.isInterleaved())

    sourceNode.initWithFormat_renderBlock_(inputFormat, self.render_block)

    audioEngine.attachNode_(sourceNode)
    sourceNode.volume = 0.1

    audioEngine.connect_to_format_(sourceNode, mainMixer, inputFormat)
    audioEngine.connect_to_format_(mainMixer, outputNode, inputFormat)

    audioEngine.prepare()
    self.audioEngine = audioEngine

  def source_node_render(self,
                         _cmd: ctypes.c_void_p,
                         _isSilence_ptr: ctypes.c_void_p,
                         _timestamp_ptr: ctypes.c_void_p,
                         frameCount: ctypes.c_void_p,
                         outputData_ptr: ctypes.POINTER) -> OSStatus:
    ablPointer = outputData_ptr.contents
    for frame in range(frameCount):
      sampleVal = self.parent.toneGenerator(self.timex)

      self.timex += self.deltaTime

      for bufferr in range(ablPointer.mNumberBuffers):
        _mData = ablPointer.mBuffers[bufferr].mData
        _pointer = ctypes.POINTER(ctypes.c_float * frameCount)
        buffer = ctypes.cast(_mData, _pointer).contents
        buffer[frame] = sampleVal
    return 0

  def start(self):
    self.audioEngine.startAndReturnError_(None)

  def stop(self):
    self.audioEngine.stop()


class View(ui.View):
  def __init__(self, *args, **kwargs):
    ui.View.__init__(self, *args, **kwargs)

    self.osc = Oscillator()
    self.setup_type_slider()
    self.toneGenerator = self.osc.wave_types[0]

    self.synth = Synth(self)
    self.synth.start()

  def setup_type_slider(self):
    self.type_len = len(self.osc.wave_types) - 1
    self.type_osc = ui.Slider()
    self.type_osc.continuous = False
    self.type_osc.value = 0
    self.type_osc.flex = 'W'
    self.type_osc.action = self.change_osc
    self.add_subview(self.type_osc)

  def change_osc(self, sender):
    val = int(sender.value * self.type_len)
    self.toneGenerator = self.osc.wave_types[val]
    self.type_osc.value = val / self.type_len
    print(self.toneGenerator)

  def will_close(self):
    self.synth.stop()


if __name__ == '__main__':
  view = View()
  view.present()
  #view.present(style='fullscreen', orientations=['portrait'])

```

##### frequency 選択

波形の変更と同様に`frequency`（周波数）もスライダーで変更できるようにしましょう。

ほぼ同様の実装内容です。`ui.Slider.continuous` をデフォルトの`True` としており、スライダー位置が動く度に数値が変化します。

`self.max_level_frq = 880.0` と最大周波数を`880.0` と決め打ちにし、`ui.Slider.value` の`0.0 〜 1.0` の間のvalue 調整をしています。

```frequencyを追加.py
from math import sin, pi
from random import uniform
import ctypes

from objc_util import ObjCClass, ObjCBlock
import ui

import pdbg

CHANNEL = 1

OSStatus = ctypes.c_int32

AVAudioEngine = ObjCClass('AVAudioEngine')
AVAudioSourceNode = ObjCClass('AVAudioSourceNode')
AVAudioFormat = ObjCClass('AVAudioFormat')


class AudioBuffer(ctypes.Structure):
  _fields_ = [
    ('mNumberChannels', ctypes.c_uint32),
    ('mDataByteSize', ctypes.c_uint32),
    ('mData', ctypes.c_void_p),
  ]


class AudioBufferList(ctypes.Structure):
  _fields_ = [
    ('mNumberBuffers', ctypes.c_uint32),
    ('mBuffers', AudioBuffer * CHANNEL),
  ]


# --- OSC
class Oscillator:
  def __init__(self):
    self.amplitude: float = 1.0
    self.frequency: float = 440.0
    self.wave_types = [
      self.sine,
      self.triangle,
      self.sawtooth,
      self.square,
      self.white_noise,
    ]

  def sine(self, time):
    wave = self.amplitude * sin(2.0 * pi * self.frequency * time)
    return wave

  def triangle(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    value = currentTime / period
    result = 0.0
    if value < 0.25:
      result = value * 4
    elif value < 0.75:
      result = 2.0 - (value * 4.0)
    else:
      result = value * 4 - 4.0
    wave = self.amplitude * result
    return wave

  def sawtooth(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    wave = self.amplitude * ((currentTime / period) * 2 - 1.0)
    return wave

  def square(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    if (currentTime / period) < 0.5:
      wave = self.amplitude
    else:
      wave = -1.0 * self.amplitude
    return wave

  def white_noise(self, _):
    return uniform(-1.0, 1.0)


class Synth:
  def __init__(self, parent):
    self.parent: ui.View = parent
    self.audioEngine: AVAudioEngine
    self.sampleRate: float = 44100.0  # set_up メソッド: outputNode より確定
    self.deltaTime: float = 0.0  # 1/sampleRate 時間間隔
    self.timex: float = 0.0  # Render の間隔カウンター

    self.render_block = ObjCBlock(
      self.source_node_render,
      restype=OSStatus,
      argtypes=[
        ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p,
        ctypes.POINTER(AudioBufferList)
      ])

    self.set_up()

  def set_up(self):
    audioEngine = AVAudioEngine.new()
    sourceNode = AVAudioSourceNode.alloc()
    mainMixer = audioEngine.mainMixerNode()
    outputNode = audioEngine.outputNode()
    format = outputNode.inputFormatForBus_(0)

    self.sampleRate = format.sampleRate()
    self.deltaTime = 1 / self.sampleRate

    inputFormat = AVAudioFormat.alloc(
    ).initWithCommonFormat_sampleRate_channels_interleaved_(
      format.commonFormat(), self.sampleRate, CHANNEL, format.isInterleaved())

    sourceNode.initWithFormat_renderBlock_(inputFormat, self.render_block)

    audioEngine.attachNode_(sourceNode)
    sourceNode.volume = 0.1

    audioEngine.connect_to_format_(sourceNode, mainMixer, inputFormat)
    audioEngine.connect_to_format_(mainMixer, outputNode, inputFormat)

    audioEngine.prepare()
    self.audioEngine = audioEngine

  def source_node_render(self,
                         _cmd: ctypes.c_void_p,
                         _isSilence_ptr: ctypes.c_void_p,
                         _timestamp_ptr: ctypes.c_void_p,
                         frameCount: ctypes.c_void_p,
                         outputData_ptr: ctypes.POINTER) -> OSStatus:
    ablPointer = outputData_ptr.contents
    for frame in range(frameCount):
      sampleVal = self.parent.toneGenerator(self.timex)

      self.timex += self.deltaTime

      for bufferr in range(ablPointer.mNumberBuffers):
        _mData = ablPointer.mBuffers[bufferr].mData
        _pointer = ctypes.POINTER(ctypes.c_float * frameCount)
        buffer = ctypes.cast(_mData, _pointer).contents
        buffer[frame] = sampleVal
    return 0

  def start(self):
    self.audioEngine.startAndReturnError_(None)

  def stop(self):
    self.audioEngine.stop()


class View(ui.View):
  def __init__(self, *args, **kwargs):
    ui.View.__init__(self, *args, **kwargs)

    self.setup_osc()

    self.synth = Synth(self)
    self.synth.start()

  def setup_osc(self):
    self.osc = Oscillator()
    self.setup_type_slider()
    self.setup_frq_slider()
    self.toneGenerator = self.osc.wave_types[0]

  def setup_type_slider(self):
    self.type_len = len(self.osc.wave_types) - 1
    self.type_osc = ui.Slider()
    self.type_osc.continuous = False
    self.type_osc.value = 0
    self.type_osc.flex = 'W'
    self.type_osc.action = self.change_osc
    self.add_subview(self.type_osc)

  def change_osc(self, sender):
    val = int(sender.value * self.type_len)
    self.toneGenerator = self.osc.wave_types[val]
    self.type_osc.value = val / self.type_len

  def setup_frq_slider(self):
    self.max_level_frq = 880.0
    self.level_frq = ui.Slider()
    self.level_frq.value = self.osc.frequency / self.max_level_frq
    self.level_frq.flex = 'W'
    self.level_frq.action = self.change_frq
    self.add_subview(self.level_frq)

  def change_frq(self, sender):
    val = sender.value * self.max_level_frq
    self.osc.frequency = val

  def layout(self):
    self.level_frq.y = self.type_osc.height

  def will_close(self):
    self.synth.stop()


if __name__ == '__main__':
  view = View()
  #view.present()
  # スライダー操作時にView が動いてしまうため
  view.present(style='fullscreen', orientations=['portrait'])

```

## 音声情報の可視化

ここまでくると、View 上部にスライダー2本のみ。という見た目が寂しくなってきました。

（GIF でも、同じ絵で押し通せなくなっています）

[AVAudioEngineを使って音声にリアルタイムにエフェクトをかけて保存する方法(サンプルあり) - Qiita](https://qiita.com/ahiru___/items/2233c23a0e8070d265ea)

https://qiita.com/ahiru___/items/2233c23a0e8070d265ea

render として自作した`block`（`source_node_render`）の音になる情報を、別の`block` を作ることで取得ができそうです。

[installTapOnBus:bufferSize:format:block: | Apple Developer Documentation](https://developer.apple.com/documentation/avfaudio/avaudionode/1387122-installtaponbus)

https://developer.apple.com/documentation/avfaudio/avaudionode/1387122-installtaponbus

`tap_block` として、`ObjCBlock` を定義します:

```py
self.tap_block = ObjCBlock(
  self.audio_node_tap,
  restype=None,
  argtypes=[
    ctypes.c_void_p,
    ctypes.c_void_p,
    ctypes.c_void_p,
  ])
```

コールバック関数として、`audio_node_tap` をつくります:

```py
def audio_node_tap(self, _cmd, buffer, when):
  # objc_util の処理
  buf = ObjCInstance(buffer)
  _array = buf.floatChannelData()[0]
  
  # ここからPython の処理
  # 画像生成用に配列組み替え
  np_buff = np.ctypeslib.as_array(_array, (256, 16))
  with BytesIO() as bIO:
    matplotlib.image.imsave(bIO, np_buff + 1, format='png')
    img = ui.Image.from_data(bIO.getvalue())
    self.parent.visualize_view.image = img
```

`objc_util` で考えることと、Python（Pythonista3）で考えることを分けて、混乱する要因を減らします。

先に`objc_util` の処理を進めます。

### `objc_util` の処理

`Synth` 内の、connect 処理をしている最終点の `mainMixer` から取得をします。

（数式で生成した情報を取りたい場合には`sourceNode`）

```py
audioEngine.connect_to_format_(mainMixer, outputNode, inputFormat)
audioEngine.prepare()

_bufsize = 64 * 64  # 取得する情報量
mainMixer.installTapOnBus_bufferSize_format_block_(
  0, _bufsize, inputFormat, self.tap_block)
self.audioEngine = audioEngine
```

`_bufsize` にて、取りたいサイズ指定をします。無駄に多くても処理が追いつかない可能性があるので、`4096 = 64*64` としました。

`installTapOnBus_bufferSize_format_block_` にて、`audio_node_tap` が呼び出されるようになりました。

```py
def audio_node_tap(self, _cmd, buffer, when):
  # objc_util の処理
  buf = ObjCInstance(buffer)
  _array = buf.floatChannelData()[0]
```

型定義を事細かく説明していましたが、`objc_util.ObjCInstance` で問題ないのであれば`ctypes.c_void_p` で定義してしまっています。

`ObjCInstance` で問題ないなら、それはそれでオッケー的な考え方です（雑）。

`floatChannelData()` は、配列で来るのでindexの`[0]` で情報を貰います。

`block` は、毎回すごい早さで呼び出されるので、`print` 確認が大変です。

実行後に即終了させて`print` 出力量を減らし、必要な情報を確認する作業をしています（もちろん事前にDocumentation で返ってくる内容を把握することも大事です）。

音データの内部構造など、めちゃくちゃ面白く深いのですが、長くなりそうなので泣く泣く割愛します。。。

#### Python の処理

`audio_node_tap` メソッドにて、Objective-C（`objc_util`）の情報を、通常のPython で処理できるようになりました。

ざっくりとした説明ですと、PCM フォーマットの音の情報（`-1.0 〜 1.0`）が配列として入っています。

最終的に`ui.Image` へ乗せたいので、バイト列の画像情報に変換をさせたいのです。

:::note warn
ここからのPython 処理
当時どのように調べ実装したのかの記録がなく
ゆるふわな、説明になってしまいます🙇
:::

##### NumPy で配列を整える

配列の形状を変更させるのは、NumPy が強いので（しかも`ctypes` 経由の操作なので）、`np.ctypeslib.as_array` で形状を整えます。

最終出力画像を確認しながらの調整ですが、`4096 = (256, 16)` が綺麗な横並びとなったので、そのような数値としています。

##### Matplotlib で画像を作る

Matplotlib でグラフは作りません。

（かつて）音データだった配列を、バイトデータとして画像化します。

```py
np_buff = np.ctypeslib.as_array(_array, (256, 16))
with BytesIO() as bIO:
  matplotlib.image.imsave(bIO, np_buff + 1, format='png')
  img = ui.Image.from_data(bIO.getvalue())

```

`PIL` モジュールでもいい気がしますが、なぜかMatplotlib を使っている理由は不明です。。。

[Python Imaging Library — Python 3.6.1 documentation](https://omz-software.com/pythonista/docs/ios/PIL.html)

https://omz-software.com/pythonista/docs/ios/PIL.html

Matplotlib の方が当時やりやすかったのかもしれません。

使いやすい、やりやすい方法で実装いただいて問題ありません。

変数`img` として、`ui.Image` に乗せられる状態になりました。

### `ui.View` での可視化

最終出力となる音データを抜き出し、画像情報に変わるよう配列を整形し`ui.Image` として`img` の準備まで完了しました。

class の`View` へ、乗せたいものを乗せていきましょう:

```py
from math import sin, pi
from random import uniform
import ctypes
from io import BytesIO

import numpy as np
import matplotlib.image

from objc_util import ObjCClass, ObjCInstance, ObjCBlock
import ui

import pdbg

CHANNEL = 1

OSStatus = ctypes.c_int32

AVAudioEngine = ObjCClass('AVAudioEngine')
AVAudioSourceNode = ObjCClass('AVAudioSourceNode')
AVAudioFormat = ObjCClass('AVAudioFormat')


class AudioBuffer(ctypes.Structure):
  _fields_ = [
    ('mNumberChannels', ctypes.c_uint32),
    ('mDataByteSize', ctypes.c_uint32),
    ('mData', ctypes.c_void_p),
  ]


class AudioBufferList(ctypes.Structure):
  _fields_ = [
    ('mNumberBuffers', ctypes.c_uint32),
    ('mBuffers', AudioBuffer * CHANNEL),
  ]


class Oscillator:
  def __init__(self):
    self.amplitude: float = 1.0
    self.frequency: float = 440.0
    self.wave_types = [
      self.sine,
      self.triangle,
      self.sawtooth,
      self.square,
      self.white_noise,
    ]

  def sine(self, time):
    wave = self.amplitude * sin(2.0 * pi * self.frequency * time)
    return wave

  def triangle(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    value = currentTime / period
    result = 0.0
    if value < 0.25:
      result = value * 4
    elif value < 0.75:
      result = 2.0 - (value * 4.0)
    else:
      result = value * 4 - 4.0
    wave = self.amplitude * result
    return wave

  def sawtooth(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    wave = self.amplitude * ((currentTime / period) * 2 - 1.0)
    return wave

  def square(self, time):
    period = 1.0 / self.frequency
    currentTime = time % period
    if (currentTime / period) < 0.5:
      wave = self.amplitude
    else:
      wave = -1.0 * self.amplitude
    return wave

  def white_noise(self, _):
    return uniform(-1.0, 1.0)


class Synth:
  def __init__(self, parent):
    self.parent: ui.View = parent  # 親となるui.View
    self.audioEngine: AVAudioEngine
    self.sampleRate: float = 44100.0  # set_up メソッド: outputNode より確定
    self.deltaTime: float = 0.0  # 1/sampleRate 時間間隔
    self.timex: float = 0.0  # Render の間隔カウンター

    self.render_block = ObjCBlock(
      self.source_node_render,
      restype=OSStatus,
      argtypes=[
        ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p,
        ctypes.POINTER(AudioBufferList)
      ])

    self.tap_block = ObjCBlock(
      self.audio_node_tap,
      restype=None,
      argtypes=[
        ctypes.c_void_p,
        ctypes.c_void_p,
        ctypes.c_void_p,
      ])

    self.set_up()

  def set_up(self):
    audioEngine = AVAudioEngine.new()
    sourceNode = AVAudioSourceNode.alloc()
    mainMixer = audioEngine.mainMixerNode()
    outputNode = audioEngine.outputNode()
    format = outputNode.inputFormatForBus_(0)

    self.sampleRate = format.sampleRate()
    self.deltaTime = 1 / self.sampleRate

    inputFormat = AVAudioFormat.alloc(
    ).initWithCommonFormat_sampleRate_channels_interleaved_(
      format.commonFormat(), self.sampleRate, CHANNEL, format.isInterleaved())

    sourceNode.initWithFormat_renderBlock_(inputFormat, self.render_block)

    audioEngine.attachNode_(sourceNode)
    sourceNode.volume = 0.1

    audioEngine.connect_to_format_(sourceNode, mainMixer, inputFormat)
    audioEngine.connect_to_format_(mainMixer, outputNode, inputFormat)

    audioEngine.prepare()

    _bufsize = 64 * 64  # 取得する情報量
    mainMixer.installTapOnBus_bufferSize_format_block_(
      0, _bufsize, inputFormat, self.tap_block)
    self.audioEngine = audioEngine

  def source_node_render(self,
                         _cmd: ctypes.c_void_p,
                         _isSilence_ptr: ctypes.c_void_p,
                         _timestamp_ptr: ctypes.c_void_p,
                         frameCount: ctypes.c_void_p,
                         outputData_ptr: ctypes.POINTER) -> OSStatus:
    ablPointer = outputData_ptr.contents
    for frame in range(frameCount):
      sampleVal = self.parent.toneGenerator(self.timex)

      self.timex += self.deltaTime

      for bufferr in range(ablPointer.mNumberBuffers):
        _mData = ablPointer.mBuffers[bufferr].mData
        _pointer = ctypes.POINTER(ctypes.c_float * frameCount)
        buffer = ctypes.cast(_mData, _pointer).contents
        buffer[frame] = sampleVal
    return 0

  def audio_node_tap(self, _cmd, buffer, when):
    buf = ObjCInstance(buffer)
    _array = buf.floatChannelData()[0]
    # 画像生成用に配列組み替え
    np_buff = np.ctypeslib.as_array(_array, (256, 16))
    with BytesIO() as bIO:
      matplotlib.image.imsave(bIO, np_buff + 1, format='png')
      img = ui.Image.from_data(bIO.getvalue())
      self.parent.visualize_view.image = img

  def start(self):
    self.audioEngine.startAndReturnError_(None)

  def stop(self):
    self.audioEngine.stop()


class View(ui.View):
  def __init__(self, *args, **kwargs):
    ui.View.__init__(self, *args, **kwargs)

    self.visualize_view = ui.ImageView()
    self.visualize_view.bg_color = 0
    self.visualize_view.flex = 'WH'
    self.add_subview(self.visualize_view)

    self.type_osc = ui.Slider()
    self.level_frq = ui.Slider()
    self.osc_log = ui.Label()
    self.frq_log = ui.Label()

    self.osc = Oscillator()
    self.toneGenerator: Oscillator
    self.setup_osc()

    self.synth = Synth(self)
    self.synth.start()

  def setup_osc(self):
    self.toneGenerator = self.osc.wave_types[0]
    self.setup_type_osc()
    self.setup_frq_level()

  def setup_type_osc(self):
    # --- slider
    self.type_len = len(self.osc.wave_types) - 1
    self.type_osc.continuous = False
    self.type_osc.value = 0
    self.type_osc.flex = 'W'
    self.type_osc.action = self.change_osc
    self.add_subview(self.type_osc)

    # --- label
    self.osc_log.text = self.toneGenerator.__name__
    self.osc_log.bg_color = 1
    self.osc_log.flex = 'W'
    self.osc_log.size_to_fit()
    self.add_subview(self.osc_log)

  def change_osc(self, sender):
    val = int(sender.value * self.type_len)
    self.toneGenerator = self.osc.wave_types[val]
    self.type_osc.value = val / self.type_len
    self.osc_log.text = self.toneGenerator.__name__

  def setup_frq_level(self):
    # --- slider
    self.max_level_frq = 880.0
    self.level_frq.value = self.osc.frequency / self.max_level_frq
    self.level_frq.flex = 'W'
    self.level_frq.action = self.change_frq
    self.add_subview(self.level_frq)

    # --- label
    self.frq_log.text = f'{self.osc.frequency}'
    self.frq_log.bg_color = 1
    self.frq_log.flex = 'W'
    self.frq_log.size_to_fit()
    self.add_subview(self.frq_log)

  def change_frq(self, sender):
    val = sender.value * self.max_level_frq
    self.osc.frequency = val
    self.frq_log.text = f'{self.osc.frequency}'

  def layout(self):
    # --- slider
    self.type_osc.y = self.type_osc.height
    self.level_frq.y = self.type_osc.height + self.type_osc.y * 2

    # --- label
    self.osc_log.y = self.frame.height / 2 - self.osc_log.height
    self.frq_log.y = self.osc_log.y + self.frq_log.height

    logs_width = self.frame.width / 2
    self.osc_log.width = self.frq_log.width = logs_width

  def will_close(self):
    self.synth.stop()


if __name__ == '__main__':
  view = View()
  #view.present()
  # スライダー操作時にView が動いてしまうため
  view.present(style='fullscreen', orientations=['portrait'])

```

![img221206_155006.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2953777/5aeb1ca5-f274-aaae-673a-20a901e3c29e.gif)



かなりfat な`View` class となってしまいました、、、

- 音声情報の可視化
  - `self.visualize_view`
- osc 関係
  - 操作
    - `self.type_osc`
  - 表示
    - `self.osc_log`
- 周波数（`frequency`）関係
  - 操作
    - `self.level_frq`
  - 表示
    - `self.frq_log`

View の画像の柄により、波形の特徴がわかります。


![img221206_155705.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2953777/4067c238-217d-c3dd-d29b-812fea1a1cc4.gif)


```
赤 → 黄 → 緑 → 青
```

の、色の移り変わりで数値の強弱が出力されているとイメージするとわかりやすいです。

#### 正弦波: sine


![img221206_155858.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2953777/9db8851c-8db7-f874-b612-b86f7c30786c.png)



綺麗に波打つ波形になるので、グラデーションの変化が滑らかに出ています。

#### 三角波: triangle


![img221206_155909.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2953777/cc7c4652-51ee-4d3c-3baa-590a73212c38.png)



下から上へ直線に`△` 勾配している状態です。

#### ノコギリ波: sawtooth


![img221206_155917.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2953777/ec15adf6-ab0f-fbd9-b035-3be449f476ca.png)



`|\` の勾配となるので、`青` から突然`赤` になっているのが確認できます。

#### 矩形波: square

![img221206_155926.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2953777/7b8b81f6-5dd3-bd83-ee2f-5a31422ffd05.png)



`_| ̄|_` と、上下間を直角に変化させるので、`青` と`赤` しか出ていません。

#### ノイズ: WhiteNoise


![img221206_155940.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2953777/2dc1e9e9-24e8-7a7e-b2a0-4d0aa5dd916a.png)



`-1.0 〜 1.0` をランダムで変化させているので、統一性がなくカラフルに出力されています。

## 作曲編曲！？波形をいじってみよう

波形同士を組み合わせることで、定型の波形から変化させることができます。

`Oscillator` class の中身を一部改変して、独自の波形を音として出力してみましょう。

変化がキツい三角波と、波形を混ぜ合わせるメソッドをつくりました:

```py
def tone_triangle(self, time, frq=None):
  frequency = frq if frq else self.frequency
  period = 1.0 / frequency
  currentTime = time % period
  value = currentTime / period
  result = 0.0
  if value < 0.0:
    result = value * 4
  elif value > 0.8:
    result = value * 4 - 4.0
  else:
    result = 0
  wave = self.amplitude * result
  return wave

def mixwave(self, time):
  _step = 3 + int(sin(pi * time) * 10)
  steps = _step if _step else 1
  wave01 = self.square(time) * self.tone_triangle(time, steps)
  wave02 = self.white_noise(time) * self.tone_triangle(time, 1)
  wave = wave01 + wave02
  return wave
```

引数に`frq=None` を付けて、`self.frequency` 以外でも`frequency` を設定できるようにしています:

```py
def tone_triangle(self, time, frq=None):
  frequency = frq if frq else self.frequency
```

`frq=None` は他の関数にも、実装させています。

`Oscillator` の配列にも登録してあげましょう:

```py
self.wave_types = [
  self.mixwave,  # new!
  self.sine,
  self.triangle,
  self.sawtooth,
  self.square,
  self.white_noise,
  self.tone_triangle,  # new!
]
```


![img221206_184633.gif](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/2953777/74926e3a-e3bc-0e51-e6de-bef451b98223.gif)



コード例では、画面の色変化のバキバキが強いので、小さくしています。

```py
from math import sin, pi
from random import uniform
import ctypes
from io import BytesIO

import numpy as np
import matplotlib.image

from objc_util import ObjCClass, ObjCInstance, ObjCBlock
import ui

import pdbg

CHANNEL = 1

OSStatus = ctypes.c_int32

AVAudioEngine = ObjCClass('AVAudioEngine')
AVAudioSourceNode = ObjCClass('AVAudioSourceNode')
AVAudioFormat = ObjCClass('AVAudioFormat')


class AudioBuffer(ctypes.Structure):
  _fields_ = [
    ('mNumberChannels', ctypes.c_uint32),
    ('mDataByteSize', ctypes.c_uint32),
    ('mData', ctypes.c_void_p),
  ]


class AudioBufferList(ctypes.Structure):
  _fields_ = [
    ('mNumberBuffers', ctypes.c_uint32),
    ('mBuffers', AudioBuffer * CHANNEL),
  ]


class Oscillator:
  def __init__(self):
    self.amplitude: float = 1.0
    self.frequency: float = 440.0
    self.wave_types = [
      self.mixwave,
      self.sine,
      self.triangle,
      self.sawtooth,
      self.square,
      self.white_noise,
      self.tone_triangle,
    ]

  def sine(self, time, *args):
    frequency = args[0] if args else self.frequency
    wave = self.amplitude * sin(2.0 * pi * frequency * time)
    return wave

  def triangle(self, time, frq=None):
    frequency = frq if frq else self.frequency
    period = 1.0 / frequency
    currentTime = time % period
    value = currentTime / period
    result = 0.0
    if value < 0.25:
      result = value * 4
    elif value < 0.75:
      result = 2.0 - (value * 4.0)
    else:
      result = value * 4 - 4.0
    wave = self.amplitude * result
    return wave

  def sawtooth(self, time, frq=None):
    frequency = frq if frq else self.frequency
    period = 1.0 / frequency
    currentTime = time % period
    wave = self.amplitude * ((currentTime / period) * 2 - 1.0)
    return wave

  def square(self, time, frq=None):
    frequency = frq if frq else self.frequency
    period = 1.0 / frequency
    currentTime = time % period
    if (currentTime / period) < 0.5:
      wave = self.amplitude
    else:
      wave = -1.0 * self.amplitude
    return wave

  def white_noise(self, _, frq=None):
    return uniform(-1.0, 1.0)

  def tone_triangle(self, time, frq=None):
    frequency = frq if frq else self.frequency
    period = 1.0 / frequency
    currentTime = time % period
    value = currentTime / period
    result = 0.0
    if value < 0.0:
      result = value * 4
    elif value > 0.8:
      result = value * 4 - 4.0
    else:
      result = 0
    wave = self.amplitude * result
    return wave

  def mixwave(self, time):
    _step = 3 + int(sin(pi * time) * 10)
    steps = _step if _step else 1
    wave01 = self.square(time) * self.tone_triangle(time, steps)
    wave02 = self.white_noise(time) * self.tone_triangle(time, 1)
    wave = wave01 + wave02
    return wave


class Synth:
  def __init__(self, parent):
    self.parent: ui.View = parent  # 親となるui.View
    self.audioEngine: AVAudioEngine
    self.sampleRate: float = 44100.0  # set_up メソッド: outputNode より確定
    self.deltaTime: float = 0.0  # 1/sampleRate 時間間隔
    self.timex: float = 0.0  # Render の間隔カウンター

    self.render_block = ObjCBlock(
      self.source_node_render,
      restype=OSStatus,
      argtypes=[
        ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p,
        ctypes.POINTER(AudioBufferList)
      ])

    self.tap_block = ObjCBlock(
      self.audio_node_tap,
      restype=None,
      argtypes=[
        ctypes.c_void_p,
        ctypes.c_void_p,
        ctypes.c_void_p,
      ])

    self.set_up()

  def set_up(self):
    audioEngine = AVAudioEngine.new()
    sourceNode = AVAudioSourceNode.alloc()
    mainMixer = audioEngine.mainMixerNode()
    outputNode = audioEngine.outputNode()
    format = outputNode.inputFormatForBus_(0)

    self.sampleRate = format.sampleRate()
    self.deltaTime = 1 / self.sampleRate

    inputFormat = AVAudioFormat.alloc(
    ).initWithCommonFormat_sampleRate_channels_interleaved_(
      format.commonFormat(), self.sampleRate, CHANNEL, format.isInterleaved())

    sourceNode.initWithFormat_renderBlock_(inputFormat, self.render_block)

    audioEngine.attachNode_(sourceNode)
    sourceNode.volume = 0.1

    audioEngine.connect_to_format_(sourceNode, mainMixer, inputFormat)
    audioEngine.connect_to_format_(mainMixer, outputNode, inputFormat)

    audioEngine.prepare()

    _bufsize = 64 * 64  # 取得する情報量
    mainMixer.installTapOnBus_bufferSize_format_block_(
      0, _bufsize, inputFormat, self.tap_block)
    self.audioEngine = audioEngine

  def source_node_render(self,
                         _cmd: ctypes.c_void_p,
                         _isSilence_ptr: ctypes.c_void_p,
                         _timestamp_ptr: ctypes.c_void_p,
                         frameCount: ctypes.c_void_p,
                         outputData_ptr: ctypes.POINTER) -> OSStatus:
    ablPointer = outputData_ptr.contents
    for frame in range(frameCount):
      sampleVal = self.parent.toneGenerator(self.timex)

      self.timex += self.deltaTime

      for bufferr in range(ablPointer.mNumberBuffers):
        _mData = ablPointer.mBuffers[bufferr].mData
        _pointer = ctypes.POINTER(ctypes.c_float * frameCount)
        buffer = ctypes.cast(_mData, _pointer).contents
        buffer[frame] = sampleVal
    return 0

  def audio_node_tap(self, _cmd, buffer, when):
    buf = ObjCInstance(buffer)
    _array = buf.floatChannelData()[0]
    # 画像生成用に配列組み替え
    np_buff = np.ctypeslib.as_array(_array, (256, 16))
    with BytesIO() as bIO:
      matplotlib.image.imsave(bIO, np_buff + 1, format='png')
      img = ui.Image.from_data(bIO.getvalue())
      self.parent.visualize_view.image = img

  def start(self):
    self.audioEngine.startAndReturnError_(None)

  def stop(self):
    self.audioEngine.stop()


class View(ui.View):
  def __init__(self, *args, **kwargs):
    ui.View.__init__(self, *args, **kwargs)

    self.visualize_view = ui.ImageView()
    self.visualize_view.bg_color = 0
    #self.visualize_view.flex = 'WH'
    self.visualize_view.flex = 'TBRL'
    self.add_subview(self.visualize_view)

    self.type_osc = ui.Slider()
    self.level_frq = ui.Slider()
    self.osc_log = ui.Label()
    self.frq_log = ui.Label()

    self.osc = Oscillator()
    self.toneGenerator: Oscillator
    self.setup_osc()

    self.synth = Synth(self)
    self.synth.start()

  def setup_osc(self):
    self.toneGenerator = self.osc.wave_types[0]
    self.setup_type_osc()
    self.setup_frq_level()

  def setup_type_osc(self):
    # --- slider
    self.type_len = len(self.osc.wave_types) - 1
    self.type_osc.continuous = False
    self.type_osc.value = 0
    self.type_osc.flex = 'W'
    self.type_osc.action = self.change_osc
    self.add_subview(self.type_osc)

    # --- label
    self.osc_log.text = self.toneGenerator.__name__
    self.osc_log.bg_color = 1
    self.osc_log.flex = 'W'
    self.osc_log.size_to_fit()
    self.add_subview(self.osc_log)

  def change_osc(self, sender):
    val = int(sender.value * self.type_len)
    self.toneGenerator = self.osc.wave_types[val]
    self.type_osc.value = val / self.type_len
    self.osc_log.text = self.toneGenerator.__name__

  def setup_frq_level(self):
    # --- slider
    self.max_level_frq = 880.0
    self.level_frq.value = self.osc.frequency / self.max_level_frq
    self.level_frq.flex = 'W'
    self.level_frq.action = self.change_frq
    self.add_subview(self.level_frq)

    # --- label
    self.frq_log.text = f'{self.osc.frequency}'
    self.frq_log.bg_color = 1
    self.frq_log.flex = 'W'
    self.frq_log.size_to_fit()
    self.add_subview(self.frq_log)

  def change_frq(self, sender):
    val = sender.value * self.max_level_frq
    self.osc.frequency = val
    self.frq_log.text = f'{self.osc.frequency}'

  def layout(self):
    # --- slider
    self.type_osc.y = self.type_osc.height
    self.level_frq.y = self.type_osc.height + self.type_osc.y * 2

    # --- label
    self.osc_log.y = self.frame.height / 2 - self.osc_log.height
    self.frq_log.y = self.osc_log.y + self.frq_log.height

    logs_width = self.frame.width / 2
    self.osc_log.width = self.frq_log.width = logs_width

  def will_close(self):
    self.synth.stop()


if __name__ == '__main__':
  view = View()
  #view.present()
  # スライダー操作時にView が動いてしまうため
  view.present(style='fullscreen', orientations=['portrait'])

```

#### エフェクトをつけよう

[AVAudioUnitDelay | Apple Developer Documentation](https://developer.apple.com/documentation/avfaudio/avaudiounitdelay?language=objc)

https://developer.apple.com/documentation/avfaudio/avaudiounitdelay?language=objc

[AVAudioUnitのパラメータ詳細（基本編） - Qiita](https://qiita.com/MJeeeey/items/e5dc75b20adb59c7d48f)

https://qiita.com/MJeeeey/items/e5dc75b20adb59c7d48f

AudioUnit のエフェクトが使えます！

呼び出しは簡単で:

```py
AVAudioUnitDelay = ObjCClass('AVAudioUnitDelay')

delay = AVAudioUnitDelay.new()
delay.delayTime = 0.3
delay.feedback = 80
```

アタッチして、Node にconnect すれば、反映されます:

```py
audioEngine.attachNode_(delay)
    
audioEngine.connect_to_format_(sourceNode, delay, inputFormat)
audioEngine.connect_to_format_(delay, mainMixer, inputFormat)

```

音がカオスなので、エフェクトがわかりにくいかもしれませんが、無事にかかっていますね！

osc の切り替えや周波数変更時が、わかりやすいかもしれません。

```py
from math import sin, pi
from random import uniform
import ctypes
from io import BytesIO

import numpy as np
import matplotlib.image

from objc_util import ObjCClass, ObjCInstance, ObjCBlock
import ui

import pdbg

CHANNEL = 1

OSStatus = ctypes.c_int32

AVAudioEngine = ObjCClass('AVAudioEngine')
AVAudioSourceNode = ObjCClass('AVAudioSourceNode')
AVAudioFormat = ObjCClass('AVAudioFormat')
AVAudioUnitDelay = ObjCClass('AVAudioUnitDelay')


class AudioBuffer(ctypes.Structure):
  _fields_ = [
    ('mNumberChannels', ctypes.c_uint32),
    ('mDataByteSize', ctypes.c_uint32),
    ('mData', ctypes.c_void_p),
  ]


class AudioBufferList(ctypes.Structure):
  _fields_ = [
    ('mNumberBuffers', ctypes.c_uint32),
    ('mBuffers', AudioBuffer * CHANNEL),
  ]


class Oscillator:
  def __init__(self):
    self.amplitude: float = 1.0
    self.frequency: float = 440.0
    self.wave_types = [
      self.mixwave,
      self.sine,
      self.triangle,
      self.sawtooth,
      self.square,
      self.white_noise,
      self.tone_triangle,
    ]

  def sine(self, time, *args):
    frequency = args[0] if args else self.frequency
    wave = self.amplitude * sin(2.0 * pi * frequency * time)
    return wave

  def triangle(self, time, frq=None):
    frequency = frq if frq else self.frequency
    period = 1.0 / frequency
    currentTime = time % period
    value = currentTime / period
    result = 0.0
    if value < 0.25:
      result = value * 4
    elif value < 0.75:
      result = 2.0 - (value * 4.0)
    else:
      result = value * 4 - 4.0
    wave = self.amplitude * result
    return wave

  def sawtooth(self, time, frq=None):
    frequency = frq if frq else self.frequency
    period = 1.0 / frequency
    currentTime = time % period
    wave = self.amplitude * ((currentTime / period) * 2 - 1.0)
    return wave

  def square(self, time, frq=None):
    frequency = frq if frq else self.frequency
    period = 1.0 / frequency
    currentTime = time % period
    if (currentTime / period) < 0.5:
      wave = self.amplitude
    else:
      wave = -1.0 * self.amplitude
    return wave

  def white_noise(self, _, frq=None):
    return uniform(-1.0, 1.0)

  def tone_triangle(self, time, frq=None):
    frequency = frq if frq else self.frequency
    period = 1.0 / frequency
    currentTime = time % period
    value = currentTime / period
    result = 0.0
    if value < 0.0:
      result = value * 4
    elif value > 0.8:
      result = value * 4 - 4.0
    else:
      result = 0
    wave = self.amplitude * result
    return wave

  def mixwave(self, time):
    _step = 3 + int(sin(pi * time) * 10)
    steps = _step if _step else 1
    wave01 = self.square(time) * self.tone_triangle(time, _step)
    wave02 = self.white_noise(time) * self.tone_triangle(time, 1)
    wave = wave01 + wave02
    return wave


class Synth:
  def __init__(self, parent):
    self.parent: ui.View = parent  # 親となるui.View
    self.audioEngine: AVAudioEngine
    self.sampleRate: float = 44100.0  # set_up メソッド: outputNode より確定
    self.deltaTime: float = 0.0  # 1/sampleRate 時間間隔
    self.timex: float = 0.0  # Render の間隔カウンター

    self.render_block = ObjCBlock(
      self.source_node_render,
      restype=OSStatus,
      argtypes=[
        ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p,
        ctypes.POINTER(AudioBufferList)
      ])

    self.tap_block = ObjCBlock(
      self.audio_node_tap,
      restype=None,
      argtypes=[
        ctypes.c_void_p,
        ctypes.c_void_p,
        ctypes.c_void_p,
      ])

    self.set_up()

  def set_up(self):
    audioEngine = AVAudioEngine.new()
    sourceNode = AVAudioSourceNode.alloc()
    mainMixer = audioEngine.mainMixerNode()
    outputNode = audioEngine.outputNode()
    format = outputNode.inputFormatForBus_(0)

    delay = AVAudioUnitDelay.new()
    delay.delayTime = 0.3
    delay.feedback = 80

    self.sampleRate = format.sampleRate()
    self.deltaTime = 1 / self.sampleRate

    inputFormat = AVAudioFormat.alloc(
    ).initWithCommonFormat_sampleRate_channels_interleaved_(
      format.commonFormat(), self.sampleRate, CHANNEL, format.isInterleaved())

    sourceNode.initWithFormat_renderBlock_(inputFormat, self.render_block)

    audioEngine.attachNode_(sourceNode)
    sourceNode.volume = 0.1

    audioEngine.attachNode_(delay)

    audioEngine.connect_to_format_(sourceNode, delay, inputFormat)
    audioEngine.connect_to_format_(delay, mainMixer, inputFormat)

    audioEngine.connect_to_format_(mainMixer, outputNode, inputFormat)

    audioEngine.prepare()

    _bufsize = 64 * 64  # 取得する情報量
    mainMixer.installTapOnBus_bufferSize_format_block_(
      0, _bufsize, inputFormat, self.tap_block)
    self.audioEngine = audioEngine

  def source_node_render(self,
                         _cmd: ctypes.c_void_p,
                         _isSilence_ptr: ctypes.c_void_p,
                         _timestamp_ptr: ctypes.c_void_p,
                         frameCount: ctypes.c_void_p,
                         outputData_ptr: ctypes.POINTER) -> OSStatus:
    ablPointer = outputData_ptr.contents
    for frame in range(frameCount):
      sampleVal = self.parent.toneGenerator(self.timex)

      self.timex += self.deltaTime

      for bufferr in range(ablPointer.mNumberBuffers):
        _mData = ablPointer.mBuffers[bufferr].mData
        _pointer = ctypes.POINTER(ctypes.c_float * frameCount)
        buffer = ctypes.cast(_mData, _pointer).contents
        buffer[frame] = sampleVal
    return 0

  def audio_node_tap(self, _cmd, buffer, when):
    buf = ObjCInstance(buffer)
    _array = buf.floatChannelData()[0]
    # 画像生成用に配列組み替え
    np_buff = np.ctypeslib.as_array(_array, (256, 16))
    with BytesIO() as bIO:
      matplotlib.image.imsave(bIO, np_buff + 1, format='png')
      img = ui.Image.from_data(bIO.getvalue())
      self.parent.visualize_view.image = img

  def start(self):
    self.audioEngine.startAndReturnError_(None)

  def stop(self):
    self.audioEngine.stop()


class View(ui.View):
  def __init__(self, *args, **kwargs):
    ui.View.__init__(self, *args, **kwargs)

    self.visualize_view = ui.ImageView()
    self.visualize_view.bg_color = 0
    self.visualize_view.flex = 'WH'
    #self.visualize_view.flex = 'TBRL'
    self.add_subview(self.visualize_view)

    self.type_osc = ui.Slider()
    self.level_frq = ui.Slider()
    self.osc_log = ui.Label()
    self.frq_log = ui.Label()

    self.osc = Oscillator()
    self.toneGenerator: Oscillator
    self.setup_osc()

    self.synth = Synth(self)
    self.synth.start()

  def setup_osc(self):
    self.toneGenerator = self.osc.wave_types[0]
    self.setup_type_osc()
    self.setup_frq_level()

  def setup_type_osc(self):
    # --- slider
    self.type_len = len(self.osc.wave_types) - 1
    self.type_osc.continuous = False
    self.type_osc.value = 0
    self.type_osc.flex = 'W'
    self.type_osc.action = self.change_osc
    self.add_subview(self.type_osc)

    # --- label
    self.osc_log.text = self.toneGenerator.__name__
    self.osc_log.bg_color = 1
    self.osc_log.flex = 'W'
    self.osc_log.size_to_fit()
    self.add_subview(self.osc_log)

  def change_osc(self, sender):
    val = int(sender.value * self.type_len)
    self.toneGenerator = self.osc.wave_types[val]
    self.type_osc.value = val / self.type_len
    self.osc_log.text = self.toneGenerator.__name__

  def setup_frq_level(self):
    # --- slider
    self.max_level_frq = 880.0
    self.level_frq.value = self.osc.frequency / self.max_level_frq
    self.level_frq.flex = 'W'
    self.level_frq.action = self.change_frq
    self.add_subview(self.level_frq)

    # --- label
    self.frq_log.text = f'{self.osc.frequency}'
    self.frq_log.bg_color = 1
    self.frq_log.flex = 'W'
    self.frq_log.size_to_fit()
    self.add_subview(self.frq_log)

  def change_frq(self, sender):
    val = sender.value * self.max_level_frq
    self.osc.frequency = val
    self.frq_log.text = f'{self.osc.frequency}'

  def layout(self):
    # --- slider
    self.type_osc.y = self.type_osc.height
    self.level_frq.y = self.type_osc.height + self.type_osc.y * 2

    # --- label
    self.osc_log.y = self.frame.height / 2 - self.osc_log.height
    self.frq_log.y = self.osc_log.y + self.frq_log.height

    logs_width = self.frame.width / 2
    self.osc_log.width = self.frq_log.width = logs_width

  def will_close(self):
    self.synth.stop()


if __name__ == '__main__':
  view = View()
  #view.present()
  # スライダー操作時にView が動いてしまうため
  view.present(style='fullscreen', orientations=['portrait'])

```

## 参考文献

- [【iOS】Core Audioでシンセサイザーを作る - Qiita](https://qiita.com/TokyoYoshida/items/df60ea8585a0223e868b)
- [Building a Synthesizer in Swift. Making audio waveforms with… | by SwiftMoji | Better Programming](https://betterprogramming.pub/building-a-synthesizer-in-swift-866cd15b731)
- [Building a Signal Generator | Apple Developer Documentation](https://developer.apple.com/documentation/avfaudio/audio_engine/building_a_signal_generator)
- [AVAudioSourceNode, AVAudioSinkNode or How I Deleted a Repo in Less Than 24 hours](https://orjpap.github.io/swift/real-time/audio/avfoundation/2020/05/19/processAudio.html)

## 次回は

2回に渡り、`AVAudioSourceNode` を使った音の生成プログラミングを行いました。

Objective-C、`objc_util`、サウンドデータ、Python モジュールと、さまざまな要素が入り混じり、大変でしたね。

このコードをベースに足したり引いたりして、サウンドプログラミングを楽しんでいただけますと幸いです。

過去に実装した、低レイヤーの音を出すリポジトリはこちらです。

[pome-ta/pysta-sine_wave001](https://github.com/pome-ta/pysta-sine_wave001)

https://github.com/pome-ta/pysta-sine_wave001

次回は、SceneKit Framework（Pythonista3 の`scene` モジュールではない）に入っていきます。

楽しい3DCG の世界が待ってますよー。

ここまで、読んでいただきありがとうございました。

## せんでん

### Discord

Pythonista3 の日本語コミュニティーがあります。みなさん優しくて、わからないところも親身に教えてくれるのでこの機会に覗いてみてください。

https://t.co/0IOW2hn1VC

### 書籍

iPhone/iPad でプログラミングする最強の本。

- [Python や Jupyter で iPhone/iPad 先端機能を簡単･自由にプログラミング！「活用篇」：hirax](https://techbookfest.org/product/1WiyV4LEev3f4YrzesRKWV?productVariantID=axX32srsSgtLtZWUtjkj99)

https://techbookfest.org/product/1WiyV4LEev3f4YrzesRKWV?productVariantID=axX32srsSgtLtZWUtjkj99

- [Python や Jupyter で iPhone/iPad 先端機能を簡単･自由にプログラミング！「土台篇」：hirax](https://techbookfest.org/product/wTZTyeibm5GQ5XgdfMrEBV?productVariantID=kRDmN1udbEYZUWbETdwL8r)

https://techbookfest.org/product/wTZTyeibm5GQ5XgdfMrEBV?productVariantID=kRDmN1udbEYZUWbETdwL8r

### その他

- サンプルコード

[Pythonista3 Advent Calendar 2022](https://qiita.com/advent-calendar/2022/pythonista3) でのコードをまとめているリポジトリがあります。

コードのエラーや変なところや改善点など。ご指摘やPR お待ちしておりますー

https://github.com/pome-ta/Pythonista3AdventCalendar2022sampleCode

- Twitter

なんしかガチャガチャしていますが、お気兼ねなくお声がけくださいませー

https://twitter.com/pome_ta93

<blockquote class="twitter-tweet"><p lang="ja" dir="ltr">やれるか、やれないか。ではなく、やるんだけども、紹介説明することは尽きないと思うけど、締め切り守れるか？って話よ！（クズ）<br><br>Pythonista3 Advent Calendar 2022 <a href="https://t.co/JKUxA525Pt">https://t.co/JKUxA525Pt</a> <a href="https://twitter.com/hashtag/Qiita?src=hash&amp;ref_src=twsrc%5Etfw">#Qiita</a></p>&mdash; pome-ta (@pome_ta93) <a href="https://twitter.com/pome_ta93/status/1588570697777152006?ref_src=twsrc%5Etfw">November 4, 2022</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>

- GitHub

基本的にGitHub にコードをあげているので、何にハマって何を実装しているのか観測できると思います。

https://github.com/pome-ta