Görüntüden müziğe dönüşüm: Yazılım aracılığıyla dijital görüntüden müzik materyali elde etme
The conversion from image to music: Obtaining musical material from digital images through software
Yazar: MEHMET CAN AYYILMAZ
Danışman: DOÇ. DR. ERAY ALTINBÜKEN ; DR. ÖĞR. ÜYESİ OZAN SARIER
İstanbul Teknik Üniversitesi / Lisansüstü Eğitim Enstitüsü / Müzikoloji ve Müzik Teorisi Ana Bilim Dalı / Müzik Teorisi ve Kompozisyon Bilim Dalı
Derece: Yüksek Lisans, 2024
Konu: Müzik = Music
Dizin: Algoritmik kompozisyon = Algorithmic composition ; Bilişimsel yaratıcılık = Computational creativity ; Sayısal müzik işleme = Digital music processing

Özet:
Bu tez çalışması, dijital görüntülerin dönüştürülmesini ve bu dönüşümün ardından
müzik besteleme sürecinde kullanılabilecek materyaller elde etmeyi hedefleyen bir
çalışmadır. Bu hedefe ulaşmak amacıyla yazılım temelli bir yöntem geliştirilmiştir.
Müzik besteleme sürecinde motif, melodi, akor gibi müzikal materyaller ortaya çıkar.
Bu önemli ögeler yani materyaller müzik kompozisyonu dahilinde belirli bir anlam
bütünlüğü sağlamak ve bir ses tasarımı oluşturmak için kullanılır. Bu çalışma resim ve
görüntüleri müzikal materyale dönüştürerek besteleme sürecinde kullanılabilecek
materyaller elde etmek için yapılmıştır. Python programlama dili kullanılarak
geliştirilen bir dizi algoritma ile dijital resim veya görüntüler MIDI aracılığıyla sese
dönüştürülmüştür. Bu yöntemle elde edilen müzikal materyaller, besteleme sürecinde
yeni, özgün ve yaratıcı materyallerin hızlı ve sistemli bir şekilde üretilmesine olanak
tanımaktadır.
Çalışmada, iki temel hipotez vardır. İlk hipoteze göre, dönüşüm algoritmasıyla işlenen
dijital görüntülerden elde edilen çıktılar, tutarlı ve sistematik bir biçimde müzikal
materyal üretebilme potansiyeline sahip olmalıdır. İkinci hipoteze göre ise dijital
görüntülerden elde edilen çıktılar, besteci müdahalesi sonrasında müzik olarak
sunulabilecek niteliğe sahip olmalıdır. Çalışmada bu iki hipoteze yönelik geliştirilen
yöntemin müzik besteleme sürecinde etkin bir rol oynayabilecek bir araç haline
gelebileceğini göstermek amaçlanmıştır.
Yöntem, dijital görüntülerden elde edilebilecek renk, renk parlaklığı, renk
doymuşluğu, piksel koordinatı gibi verileri müzik ögeleriyle eşleme üzerine
geliştirilmiştir. Müziğin temel bileşenlerinden biri olan nota, görülebilir ışık tayfındaki
renklerin frekanslarıyla eşlenip, daha sonra bu renklerin ekranlarda kullanılan RGB
renk modeli yaklaşıkları ile kullanılmıştır. Bu işlem ile bir oktav genişliğinde nota
eşleri elde edilmiştir. Birden fazla oktav kullanımı için görüntüden elde edilen
pikseldeki renk, 256 ton içerecek şekilde kurgulanan monokrom tabloya göre siyah
beyaza çevrilmiştir. Bu monokrom dizi siyahtan beyaza piyanonun ses aralığına göre
oktavlara bölünmüştür ve böylelikle pikselin siyah beyaz halinin durumuna göre
notaya oktav atanmıştır. Dönüşümden elde edilen notaların ses gürlüğü için pikseldeki
rengin renk doymuşluğu değeri kullanılmıştır ve MIDI velocity değerine
dönüştürülmüştür. Nota süresi hesaplamaları için bir yöntem teoride kurgulanmıştır
fakat uygulama aşamasında alınan sonuçlar başarılı sayılacak kadar yeterli olmamıştır.
Nota süresi hesaplaması aynı satırda aynı renge sahip ardışık piksellerin bütün olarak
tanımlanmasıyla yapılmıştır. Çalışmanın en önemli ve en ayrıştırıcı özelliklerinden
biri çıktı sonuçlarının çoksesli olmasıdır. Görüntüler soldan sağa taranırken her bir
satır ayrı bir çizgide ilerleyen nota silsilesi olarak düşünülmüştür. Bu da elde edilen 
çıktılarda teorik olarak görüntünün yüksekliği kadar farklı yatay yapıların
oluşabilmesine olanak tanımaktadır.
Dönüşümden geçen görüntülerden elde edilen çıktılar incelenirken, bir müzik eserinde
bulunabilecek ögelere benzer yapılar aranmıştır. Bunlar; bas partisi, melodik
hareketler, akorlar, arpejler gibi ögelerdir. Çıktılarda görülen küme haldeki yapılar
incelenerek potansiyel materyalleri ortaya çıkartmak için öneriler de verilmiştir. Elde
edilen çıktılarda görüntüden görüntüye miktarları değişkenlik gösterse de bahsi geçen
ögeler bulunmuştur ve gösterimi yapılmıştır.
Çıktılardaki materyallerin doku içinde işitsel olarak rahat ayırt edilmesi ve müzik
olarak kullanılabilirlik potansiyelini azami oranda görebilmek için piyanonun
homojen ses yapısı dışında bir şekilde sunulması gerektiği düşünülmüştür ve bu
yüzden bir orkestrasyon işlemi de yapılmıştır. Orkestrasyon işlemi yapılırken çıktının
orijinal ham haline olabildiğince sadık kalmak için ekstra nota eklenmemiştir.
Çalışmanın ikinci hipotezinde müzik olarak sunulmaya uygun durumda olma koşulu
vardır. Uygun olma durumu dönüşüm çıktısının el ile işlenebilirliği üzerinedir ve bu
konuda nota sayısı önemli bir etken olmuştur. Çalışma içerisinde işlenebilme durumu
detaylı şekilde açıklanmıştır. Sonuç olarak ikinci hipoteze belirli kısıtlamalar ve
varsayımlarla cevaplar verilmiştir. Elde edilen sonuçların, bilgisayarlı müzik
oluşturma ve müzikte yaratıcılık hususunda yeni araştırmalara ve yeni araçlara ilham
verme potansiyeline sahip olduğu düşünülmektedir. Bilgisayar bilimi, görsel sanatlar,
psikoloji, müzik, eğitim bilimleri gibi alanlar dahil edilerek bu çalışmayı temel alan
veya bu çalışmadan ilham alan birçok yeni araştırma ortaya çıkabilir. Araştırmanın
gidebileceği potansiyel yönlerden sonuçlar bölümünde bahsedilmiştir.

Abstract:
This thesis explores the conversion of digital images to derive materials usable in the
process of music composition through developing a software-based method. In the
process of music composition, elements such as motifs, melodies, and chords emerge.
These crucial components are utilised within the context of the music composition to
establish a specific coherence of meaning and to craft a sound design. This study is
conducted to convert images into musical materials, with the intention of obtaining
materials applicable in the composition process. Using a series of algorithms
developed with the Python programming language, digital images are converted into
sound through MIDI. This method makes possible the rapid and systematic generation
of new, original, and creative materials suitable to be used in the composition process.
The study addresses two fundamental hypotheses. According to the first hypothesis,
musical materials can be systematically produced from the outputs of digital images
processed by the conversion algorithm. According to the second hypothesis, these
outputs, possess the qualities required to be presented as music after composer
intervention
The method developed in this study involves mapping data from digital images, such
as colour, brightness, saturation, and pixel coordinates, to musical elements. The
fundamental basis of the conversion method lies in the mapping of the frequencies of
colours in the visible light spectrum to sound frequencies. Since the study involves
digital images, it has been necessary to map the frequencies of light to colour models
used in digital environments. For this mapping process, the "Approximate RGB Values
for Visible Wavelengths" algorithm created by Dan Bruton in Fortran programming
language is employed. Although Bruton's algorithm provides approximate values for
the RGB colour model, as the exact equivalence cannot be achieved, it serves the
purpose well. After obtaining RGB values, the next step involves mapping these values
to sound frequencies. The wavelengths of visible light are converted to frequency
values, represented in Terahertz (THz). To ensure that the frequencies are within the
audible range, the frequency values are reduced by dividing them by 2 until they are
equal to or less than 440 Hz, which corresponds to the fourth octave A note. The lowest
frequency value obtained is then matched to the nearest note in the twelve-tone equal
temperament system, resulting in a one-octave scale. One noteworthy modification to
this algorithm that is done in this study is the blending of colours between adjacent
octaves, specifically between the 4th octave F and the 5th octave F. This blending is
applied to mitigate the potential harshness in transitions between octaves due to colour
differences. The resulting colour is a combination of the individual RGB values,
normalised by the total number of colours involved in the blending process.This thesis explores the conversion of digital images to derive materials usable in the
process of music composition through developing a software-based method. In the
process of music composition, elements such as motifs, melodies, and chords emerge.
These crucial components are utilised within the context of the music composition to
establish a specific coherence of meaning and to craft a sound design. This study is
conducted to convert images into musical materials, with the intention of obtaining
materials applicable in the composition process. Using a series of algorithms
developed with the Python programming language, digital images are converted into
sound through MIDI. This method makes possible the rapid and systematic generation
of new, original, and creative materials suitable to be used in the composition process.
The study addresses two fundamental hypotheses. According to the first hypothesis,
musical materials can be systematically produced from the outputs of digital images
processed by the conversion algorithm. According to the second hypothesis, these
outputs, possess the qualities required to be presented as music after composer
intervention
The method developed in this study involves mapping data from digital images, such
as colour, brightness, saturation, and pixel coordinates, to musical elements. The
fundamental basis of the conversion method lies in the mapping of the frequencies of
colours in the visible light spectrum to sound frequencies. Since the study involves
digital images, it has been necessary to map the frequencies of light to colour models
used in digital environments. For this mapping process, the "Approximate RGB Values
for Visible Wavelengths" algorithm created by Dan Bruton in Fortran programming
language is employed. Although Bruton's algorithm provides approximate values for
the RGB colour model, as the exact equivalence cannot be achieved, it serves the
purpose well. After obtaining RGB values, the next step involves mapping these values
to sound frequencies. The wavelengths of visible light are converted to frequency
values, represented in Terahertz (THz). To ensure that the frequencies are within the
audible range, the frequency values are reduced by dividing them by 2 until they are
equal to or less than 440 Hz, which corresponds to the fourth octave A note. The lowest
frequency value obtained is then matched to the nearest note in the twelve-tone equal
temperament system, resulting in a one-octave scale. One noteworthy modification to
this algorithm that is done in this study is the blending of colours between adjacent
octaves, specifically between the 4th octave F and the 5th octave F. This blending is
applied to mitigate the potential harshness in transitions between octaves due to colour
differences. The resulting colour is a combination of the individual RGB values,
normalised by the total number of colours involved in the blending process.
An essential feature of this study is the polyphonic nature of the converted outputs.
The resulting polyphonic outputs can be adapted to create diverse compositions
suitable for different musical genres and instruments. To facilitate the use and
arrangement of polyphonic outputs, the twelve-tone equal temperament system was
chosen as the tuning system. This system was selected due to its standardised and
widespread use in polyphonic harmonic systems and many Western instruments.
However, it's crucial to note that other tuning systems could also be employed.
Microtonal systems, for example, could provide access to more intervals and more
colours by using smaller steps. Nevertheless, the focus on polyphony led to the initial
choice of the twelve-tone equal temperament system.
In the previous mapping process, the colours are mapped from the note A to G♯/A♭
within one octave. To allow the use of multiple octaves, it was considered essential for
a data obtained from a digital image to play a role in distributing the notes across
different octaves. For octave distribution, a relationship is established between the
brightness of colours and the octave numbers of notes. For this purpose, a
monochromatic scale encompassing black, white, and grayscale tones is created within
the RGB colour model. This colour scale, extending from (0, 0, 0) black to (255, 255,
255) white, includes a total of 256 elements or colours and is divided into parts and
mapped to octaves. In the first stages of the study, the piano was chosen to render
sound outputs. With its standard 88 keys, the piano offers a broad pitch range. The
piano's capability to play simultaneously across multiple octaves allows for the
creation of complex harmonies.
Note dynamics, affecting the loudness and forming a crucial element in musical
expression, are mapped to the saturation values extracted from the image. As a
representation of colour intensity, saturation values serve as a key parameter in shaping
the dynamics of the musical output. The algorithm converts these saturation values
into MIDI velocity values, influencing the strength and intensity of each note during
MIDI playback.
The calculation of note duration in the algorithm is based on the assumption that
consecutive pixels on the same horizontal plane, with the same MIDI note value and
identical velocity values, are treated as a single musical note, effectively representing
a sustained occurrence. For example, if two consecutive eighth notes share the same
MIDI note value and velocity value, they will be considered as a unified quarter note
in terms of duration.
While scrutinising the outputs obtained from processed images, structures akin to
elements found in a musical composition were sought. These elements include
components such as bass lines, melodic lines, chords, and arpeggios. Clusters of
structures identified in the outputs have been examined, and recommendations are
provided to unveil potential materials. It has been observed that although the quantities
of these elements vary across different outputs, they are consistently present, and their
representations are illustrated.
In addressing the second hypothesis, which posits that the musical outputs obtained
from digital images are suitable for presentation as music after composer intervention,
the focus shifts towards the refinement of these outputs. In order to facilitate a more
perceptible differentiation of the materials within the outputs and to ascertain their
usability in a musical context, it was deemed necessary to present them in a manner
distinct from the homogenous sound of the piano. Subsequently, an orchestration
process is implemented, aiming to present the materials in a way that can be more
easily discerned by the ear and be utilised in musical compositions.
The orchestration process assumes a role in elevating the overall quality of the musical
materials without the adding any extra notes. This stage is essential as it focuses on
enhancing the raw musical output in terms of tonal balance, dynamics, and overall
coherence. Importantly, the orchestration approach is carefully crafted to keep the
fundamental essence of the original output intact. The objective is not to dilute or alter
the core of these materials but rather to augment and enrich them in a manner that
aligns with established principles of musicality. This preservation of the
algorithmically generated core ensures that the intrinsic qualities of the digital imagederived musical elements persist, maintaining their authenticity amid the refining
process. The second hypothesis asserts that, following intervention by a composer, the
musical outputs derived from digital images are poised for presentation as music. The
orchestration process becomes instrumental in realising this potential by refining the
outputs and enhancing their expressive qualities for a more compelling musical
presentation. This affirmation is responded to based on suppositions in the thesis,
where the outcomes of the orchestration process not only responds to the hypothesis
but also showcase the successful conversion of generated musical materials into
potentially coherent compositions under the guidance of a composer's intervention
The findings of this study serve as a foundational source for future research in the
domain of computer-generated music and creative processes within music
composition. The method demonstrated in this study not only showcases its current
application but also lays the groundwork for the exploration of innovative techniques
and tools in subsequent research. A crucial suggestion emanating from this study is the
imperative to cultivate interdisciplinary collaboration among fields such as computer
science, visual arts, psychology, music, and education sciences. Through
collaboration, novel methods may be discovered, leading to a greater awareness of the
synthesis of visual and auditory elements. Moreover, the study posits potential
applications of its results in various domains, including live music performances, realtime music or image generation, and the integration of visual and auditory experiences.
The impact of the developed method extends beyond the confines of the research
study, indicating its versatility and potential to inspire further exploration in the realm
of computer-generated music. Researchers and practitioners are strongly encouraged
to push the boundaries of experimentation by manipulating different parameters and
variables, thus revealing unique possibilities for artistic expression. The implications
of the study extend into broader fields such as digital art and music, suggesting that its
findings may significantly contribute to the evolving landscape of creative practises.
The transformative nature of the method holds promise for influencing the utilisation
of digital images as rich sources of musical inspiration. In essence, the implications
and future directions elucidated by this study serve to position it not solely as an
isolated exploration, but rather as a pivotal catalyst for subsequent research,
collaborative initiatives, and advancements at the convergence of technology, art, and
music.