DALL·E by OpenAI: Creating
Images from Text | by Merzmensch | Towards Data Science

It’s the 5th of January 2021, and OpenAI reports back already with another AI breakthrough. 2020 they impacted the AI world at least with two Milestones: GPT-3, a powerful Natural Language Processing model, and a music generator JukeBox.
Driven by Transformer, these two approaches in various media created coherent and convincing outputs — as texts and as music. OpenAI also experimented with visual media — e.g., within ImageGPT.
Now a new star arises on the horizon of a generative AI approach: DALL-E. Called after surrealist Salvador Dalí and the SciFi animation “Wall-E”, this transformer language model represents a multimodal neural network —
AI systems that learn about concepts in several modalities, primarily the textual and visual domains, in order to better understand the world. (OpenAI)
I suppose this is a good way to avoid AGI claims. Even OpenAI is aware, that the way to AGI is long (as these stated after they experienced the hype around GPT-3):
Indeed, AGI needs an omnipotent and multimedia approach. But DALL·E is a huge step towards this.
DALL·E does what it is described to do: creating images from text prompts. It’s a 12-billion parameter of GPT-3 and can generate images from texts using text-image pairs.
Transformer-driven Neural Networks are already an overwhelming phenomenon. Even if we can discuss “comprehension”, the “understanding” of concepts is unbelievably vital. Remind the GPT-3 written “love letter by a toaster”:
In this case, GPT-3 understood:
The essence of love letters
Functionality and features of a toaster
combination of both concepts into one “Gesamtkunstwerk”
With your Prompts, you don’t just “program” transformer driven model. You transmit your request, like to a design agency — and the results may amuse, attract, or annoy you (it’s a matter of the taste).
DALL·E fulfills its job perfectly — when we may apply the adjective “perfect” to the creative approach.
In the Blog presenting DALL·E you can see the model in action — with pre-rendered images of various contexts. You can in an interactive mode even chose between some predefined parameters.
Here is, for example, a Prompts:
A [store front] that has the word [openai] written on it
Changing the second parameter (word) to “Skynet” we get another set of images:
Or: CLIP
You see, how well the task is received and realized — in various styles and visuals, based on the pre-trained model.
There were indeed another approaches for Text2Img, but DALL·E wins in every comparison:
Experimenting with objects, shapes, and colors demonstrates the capability of this Neural Network.
Using Prompts
an [armchair] in the [shape] of [avocado/pikachu/japanese mapple]
you get photorealistic imagery with an astonishing style coherence, and variety within one topic.
Interestingly, DALL·E works perfectly with zero-shot visual reasoning without additional training.
Here is a more sophisticated prompt using text and image:
You see the same cat, the postage stamp, and a variety of colors and designs.
Apropos design, the system can recognize styles and emotional sentiment of image characterization.
For example, let’s take an owl as an emoji.
Here is a confused owl:
A depressed owl:
A happy owl:
A sleepy owl:
DALL·E has also pre-training based knowledge of temporal styles, for example, a visual representation of a device from various epochs:
Even geographical and cultural knowledge is present here. For example, the system can generate images of landmarks — combined with an image prompt for style transfer:
Here you can see various SF MOMA hallucinations with the “night sky” image prompt.
Indeed, even if AGI is still in the future, DALL·E is undoubtedly a giant leap towards it — understanding the human requests, applying own knowledge, generating photorealistic and unique imagery, it opens new ways for various approaches in Art, Design, Storytelling.
Now I am eager to play around with it as soon as it is available. Because already knowing GPT-3 ability, I can freely say — it’s a mind-blowing experience.